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I. INTRODUCTION 


This report presents results achieved during the July 1, 1975, through 
October 31, 1976, perxod on the application of OP-SAPs (operational sampled 
analog processors) in pattern recognition systems. The investigations xn- 
cluded four areas: (1) human face recognition! (2) a high-speed program- 

mable transversal filter system; (3) discrete word (speech) recognition; 
and (4) a resolution enhancement ^system. 

Detailed informatxon xs avaxlable in published journal articles (see 
Appendxx) , masters theses, and a doctoral dissertation to be completed 
shortly. Thus, this report summarizes those research projects that are 
documented elsewhere. In particular, the followxng publications are based 
on this research: 

1. E. A. Parrxsh, Jr. , and E. S. McVey, "Implications of Charge- 
Coupled Devices for Pattern P,ecognition," IEEE Trans . on Comp . , 
pp. 1146-52, Nov. 1976. 

2. E. S. llcVey and E. A. Parrxsh, Jr- , "Charge-Coupled Devxces 
Applied to Control and Pattern Recognxtion Systems," Mundo 
Electronxco , Dec. 1976. 

3. E. S. McVey and E, A. Parrxsh, Jr. , "The Applicatxon of Charge- 
Coupled Devxce Processors in Automatxc Control Systems," submitted 
to IEEE Trans, on Auto. Cont. 

4. E. W. Evers, "Applicatxon of CCDs to Fourier Transform Weighted 
Transversal Filters," M. S. Thesis, University of Virginia, Sept. 
1976. 

5. C. S. Dreher, "Human Face Identification Using CCD Discrete 
Fourier Transform Filters," M. S, Thesis in preparation. Univer- 
sity of Virginia, Jan. 1977. 

6. R. Scher, "A Programmable Discrete Analog Processor for Pattern 
Recognxtion," Ph.D. Dissertation in preparation. University of 
Virginia, May 1977. 

The last two publications will contain additional results obtained after 
expiration of the present grant. 
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II. HUMAN FACE IDENTIFICATION 


Introduction 

Of partxcular interest in pattern recognition studies is the discrete 
Fourier transform and an analogous transform, the Chirp [5] z-transform [1], 
[2], [16], Implementation of these transforms has, up tintil recently, been 
restricted to digital computer simulation via the FFT (too slow for real- 
time processing) or dedicated array processors (very expensive) [3]. Charge- 
coupled devices promise to circtonvent both these difficulties [17] by offering 
real-time array processing at chip-set prices. 

The theoretical use of transversal filters and transforms has been 
discussed in the literature [4], [6], [14]. A working charge-coupled 
Chirp z-transform device has also been fabricated and tested [7]. While 
this device is a laboratory model only, the path is clearing for commercial 
manufacture of CCD building blocks which perform standard transform and 
filter operations [8]. 

The pattern recognition system described herein employs the discrete 
Fourier transform (DFT) . The brief description which follows is intended 
as a refresher on the DFT and should help to make the included graphs easier 
to interpret [9], [10], [11]. 

1. The DET is a sampled data system and, as such, imposes a 
finite, sampled record-length restriction on the data. The 
effect of this is to presuppose periodic data (Fig. 2-la) . 

2. Because of the implied periodicity of the DFT, correlation (and 
convolution) requires augmenting the data record with zeros to 
twice the original record size (Fig. 2-lb). 

3. The DFT is symmetric on the frequency axis (Fig. 2-lc) . For 
example, for a 64-point transform 

fg = DC term 

f^ “ 1st harmonic = fg^ 
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li'lgure 2-La. Implied Periodicity of DPT 




CORRECT CORRELATION INCORRECT COEHEIATION CORRECT CORRELATION 



Figure 2~lb. Zero Augmentation for Correct Correlation 




Hormalxzed Cross Correlation 

The process employed with this pattern recognition system xs that of 
normalized cross correlatxon. Given a two-dimensional pxcture, g(x,y), and 
a template, t(x,y), a template matching scheme can be defxned as 

E(m,n) = (E S(g(i,j) - t(i-m, j-n))^) (2-1) 

i j 

Removxng the square root and expanding 

E^(m,n) = E E g^(x,j) - 2g(x,3)t (x-m, 3 -n) + t^(i-ra,j-n)) (2-2) 

i 3 

Note that, for any template, t^(i-m,j-n) is constant and, hence, can be 
subtracted from the xndex E^(m,n) with no loss of information. The term 
g^(i, 3 ) is defined as the picture energy. If the picture energy varies 
only slightly over each scene, it, too, can be subtracted. Definxng the 
resultant expression as the cross correlation between scene g(x,y) and tem- 
plate t(x,y), we arrive at 

R (m,n) = E E g(x,j )t(x-m, 3 *-n) (2-3) 

1 3 

and the normalized cross correlation as 

N (m,n) = R (m,n)/(E E g(x,j)^)^/^ (2-4) 

6^ X 3 

The normalxzed cross correlation coefficient is then defxned as 

(N ^(m,n)) = N ^(T ,T ) 

' " g^t x’ y 

max over m,n (2-5) 

Thxs may also be computed via transform techniques as 

N (m,n) = F-HG(f ,f )T(f ,f ))/(E Eg(x,3)^)l/2 (2-6) 

gu * y ^ y 

where E“^ implxes the inverse Fourier transform, 6(f ,f ) xs the Fourier 

X y 
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transform of g(x,y), and T(f^,f^) = F(t(-x,-y)), i.e., the data is flipped 
about a dxagonal axis prior to being transformed [12] , 


Generalized Recognition System Using N . (T ,T ) 


Given a set of samples {g^ (x,y) • • • gj^(x,y) }, a normalized cross correla- 
tion recognition system could be structured as in Fig. 2-2. Before a system 
such as this could be configured, it would be necessary to answer the follow- 
ing questions: 


1. How much data is necessary to establish a good template match? 
(i.e. , what is the two-dimensional information bandwidth?) 

2. Is the criterion satisfied that I Z g(i»j) be approximately 

i j 

constant over all scenes? 

3. VJhat degree of accuracy can be expected? 

Bandwidth 


To investigate question 1 it was decided to first obtain as much data 
as possible in order to observe the widest possible information bandwidth, 

A data acquisition system which sampled facial scenes taken from a closed 
circuit TV camera was employed. The sampling frequency was set at 203 KHz, 
'!7hich resulted in eight samples per video line. A data record of 230 lines 
was established, giving a total of 1,840 data points. The data was quan- 
tized via a 10-bit A/D converter. To satisfy the Nyquist sampling criterion 
the video was prefiltered to 100 KHz bandtrildth. 

A two-dimensional discrete Fast Fourier Transform DFFT) of a typical 
facial scene is sho^m in Fig, 2-3. (Note the symmetry along the horizontal 
and vertical lines, as mentioned previously. The horizontal axis contains 
eight spatial frequency components, while the vertical contains 256. Fold- 
over occurs at the fifth and 129th components, respectively.) As is evident, 
the power spectrum in the vertical dimension levels off after approximately 
32 spatial harmonics, (Three dB down occurs after the third harmonic.) Thus, 
we can conclude that facial scenes are essentially composed of low spatial 
frequencies. 
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Figure 2- A Normalized Gross Correlation Recognition System 
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DC T£Rrj SOPPPESSED TO EMPHASIZE HIOHEP SPATIAL FREOUEHCV TERTIS 


■Figure 2-3. Facial Fewer Spectrum (8 x 256) 


To further investxgate the power spectrum the scene was augmented by 
adding zeros in the horizontal direction, while averaging together every 
four lines of data in the vertical. (This averaging process can be performed 
with almost no loss in information, because averaging is akin to low-pass 
filtering and will thus perserve the low spatial frequencies already deter- 
mined to contain the most power.) Augmenting the data horizontally increases 
the frequency resolution m that direction. The result is shown in Fig. 2-4. 

Low frequency terms are most apparent in Fig. 2-4. This has decidedly 
favorable practical aspects, since it implies the sampling rate can be re- 
duced (and likewise, the number of samples necessary to maintain the power 
bandwidth), while retaining most of the information. Thus, slower and less 
expensive devices with less memory are required to adequately define a 
sampled facial scene. 

Scene Power 

The criterion of question 2 was investigated by finding (E S g(i,j)^)^/^ 

i 3 

over a data base of 40 facial scenes. The results are given in Table 1. 

The average power in each scene (10-bit data) is 15233.74, with a 
standard deviation of 659.06 or approximately four perceht^f the total 
power and thus satisfies Criterion 2. 

System Implementation 

To judge actual recognition performance a system was simulated. The 
system flow chart is shovm in Fig. 2-5. A detailed description follows; 

1. FILE NAMES - All data files are placed on magnetic disk for 
quick access. Each file is assigned a number and the prefix AA 
for ease of identification. 

2, NUMBER OF BITS - An option is given as to how many bits of the 
10-bit (1,024 gray levels) words are to be used. The effect of 
using fewer bits is to increase the coarseness of the gray 
level quantization, by right- shifting the data word (10-N) 
bits, where N = bit specification (Fig. 2-6). 
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Figure 2-4, Facial Fewer Spectrum (16 x 128) 



TABLE 1 


Facial Scene Power Cg(l,j)^)^/^ for 40 Scenes 



PWTOT 

5089.6904 
5083.7793 
5106.3447 
5116.9551 
5117.9023 
5514.4854 
5540.0029 
5544.0469 
5545.9824 
5517.5703 
5498.1279 
5496.8848 
5520.9482 
5535.4053 
5533.9707 
5276.5439 
' 5373.0781 
5378.2656 
5379.6807 
5403.1885 
5402.7236 
5444.0879 
5443.8291 
5444.0176 
5458.0986 
5775.7715 
5778.9033 
5784.4375 
5799.3320 
5796.5420 
5618.2529 
5576.2373 
5601.2109 
5632.8457 
5635.1475 
5500.3447 
5469.0273 
5461.3760 
5459.4414 
5425.5361 
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Figure 2-5. System Flow Chart 
































3. CONDENSE AND/OR CONDENSE FOR CORRELATION - As has been previ- 
ously mentxoned, to perform correlation, the data must be zero 
filled in each direction to prevent wrap-aroimd errors. A 
scene of 8 s 230 lines is first augmented with zeros to 8 x 256 
to establish a modulo 2 radix in each direction. It is then 
augmented to 16 x 256 to provide zeros in the horizontal 
direction. Since the array handling size of the computer was 
effectively limited to 2^^ complex data points, it was neces- 
sary to average every four lines together to reduce the matrix 
before zero augmenting in the vertical direction. (As men- 
tioned in the discussion on information bandwidth, the effect 
of averaging is negligible.) The data was then vertically aug- 
mented to 16 X 128. See Fig. 2-7a. 

The option is also provided to further reduce the data in the vertical 
direction by averaging together a specified niimber of lines. For example, 
specifying eight lines/average after the data has already been zero aug- 
mented for correlation would result in the data structure shorn in Fig. 2-7b. 

4. VIINDOW - A choice of three window fimctions is provided [13]; 

(1) Square window 

(2) Hamming window, w(t) = .54 + .46 cos [t] < ^ 

(3) Hanning window, w(t) = .5 + .5 cos |tj < y 


It should be noted that the windows are placed over only the 
measured data - not the zero-augmented sections. 

5. FLIP - An option is provided to perform the operation 

A 

t(x,y) = t(-x,-y). Note that correlation performed without 

this operation results in two-dimensional convolution, 

F“^(G(f ,f )T(f ,f )) = g(x,y)*t(x,y) . After the FLIP option, 

X y X y 

the DFFT is taken on the available data base. 

6. MAG, PEAK, OR NONE - An output listing can be obtained of the 
entire power spectrtm (MAGnitude) , the highest value of the 
magnitude (PEAK) , or the output can be suppressed (NONE) . The 
scene power is also listed. 
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Figure 2-7. Augmenting 



(b) 

Condensing Data for Correlation, 



7 . STORE FOR EXTERNAL USE - The transformed data can be stored 
under an arbitrary file name for later use. 

8. FILTER - An option is provided to draw (via a Tektronix 4014-1 
graphics interface) any two-dimensional filter. The filter's 
magnitude components are drawn in the frequency domain and then 
stored for future use. An example of a two-dimensional filter 
IS shown in Fig. 2-8 and 2-9. Note that the filter is symmetric 
in frequency about the foldover point. 

9. STORE FOR CORRELATION — The transformed data is stored in a 
reserve file to be used in all subsequent correlations and 
convolutions as a template. 


10. CORRELATE — Correlation is performed by multiplying the stored 

transform (template T(f ,f ) or T(f ,f )) with the current 

X y X y 

transform G(f^,f^) and then performing an inverse transform. 

11. CORRELATION CONDENSED — This is a non-optional branch based 
upon whether or not the data was zero augmented for correlation. 
If not augmented, no correlation is performed; and the program 
returns to the sequence FILE NAME. 


Example 

1. Establish a template. (In all tests file AAl was used as the test 
template. See Fig. 2-10 for a listing of the template-forming sequence.) At 
this point, a file is stored which consists of a scene (8 x 230) augmented and 
reduced to 16 x 16, flipped, and transformed. 


2. Establish a sequence for template matching (Fig. 2-11). All solid 
lines indicate an established flow chart path, with the branch decisions 
printed next to the branch questions. 


3. Enter a list of files to be used in the template matching procedure. 

A typical output appears in Table 2. PT-JTOT is the scene power, while 
XHI is the peak value of the correlation. 


Results 

A brief note is necessary concerning the scene data. Scenes AAl through 
AA5 are all facial pictures of the same person, taken at different times. The 
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Figure 2-8. Two Dimensional Filter Characteristic Projections. 
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Figure 2-9. Two Dimensional Filter, Frequency Domain. 
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Figure 2-10. Template Forming Sequence. 
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Figure 2-11, Flow Chart Training Sequence 



Intermediate Output 


JJSl (Ml) 


BITS 

CORR. COND? 

LINES/AVE 

ETENDOW 

FILTER? 

PWTOT 

XHl 

10 

Y 

8 

1 

N 

5089.6904 

5089.6914 

10 

Y 

8 

1 

N 

5083.7793 

5089.5762 

10 

Y 

8 

1 

N 

5106.3447 

5089.5215 

10 

Y 

8 

1 

N 

5116.9551 

5089.4414 

10 

Y 

8 

1 

N 

5117.9023 

5089.3604 

10 

Y 

8 

1 

N 

5514.4854 

5073.4668 

10 

Y 

8 

1 

N 

5540.0029 

5073.0771 

10 

Y 

8 

1 

N 

5544.0469 

5073.4121 

10 

Y 

8 

1 

N 

5545.9824 

5073.0889 

10 

Y 

8 

1 

N 

5517.5703 

5074.0771 

10 

Y 

8 

1 

K 

5498.1279 

5074.3711 

10 

Y 

8 

1 

N 

5496.8848 

5075.0762 

10 

Y 

8 

1 

N 

5520.9482 

5074.1221 

10 

Y 

8 

1 

N 

5535.4053 

5073.4385 

10 

Y 

8 

1 

N 

5533.9707 

5073.7480 

10 

Y 

8 

1 

N 

5276.5439 

5066.8555 

10 

Y 

8 

1 

N 

5373.0781 

5079.1094 

10 

Y 

8 

1 

N 

5378.2656 

5079.4326 

10 

Y 

8 

1 

N 

5379.6807 

5075.3193 

10 

Y 

8 

1 

N 

5403.1885 

5076.4043 

10 

Y 

8 

1 

N 

5402.7236 

5078.6270 

10 

Y 

8 

1 

N 

5444.0879 

5079,0234 

10 

Y 

8 

1 

N 

5443.8291 

5078.9687 

10 

Y 

8 

1 

K 

5444.0176 

5078.3379 

10 

Y 

8 

1 

N 

5458.0986 

5078.7969 

10 

Y 

8 

1 

N 

5778.9033 

5047.7354 

10 

Y 

8 

1 

N 

5784.4375 

5046.9355 

10 

Y 

8 

1 

N 

5799.3320 

5044.3105 

10 

Y 

8 

1 

N 

5796.5420 

5043.0586 

10 

Y 

8 

1 

N 

5618.2529 

5081.7568 

10 

Y 

8 

1 

N 

5576.2373 

5079.3525 

10 

Y 

8 

1 

N 

5601.2109 

5079.8613 

10 

Y 

8 

1 

N 

5632.8457 

5081.7168 

10 

Y 

8 

1 

N 

5635.1475 

5081.5215 

10 

Y 

8 

1 

N 

5500,3447 

5012.7930 

10 

Y 

8 

1 

N 

5469.0273 

5019.4707 

10 

Y 

8 

1 

N 

5461.3760 

5021.7598 

10 

Y 

8 

1 

N 

5459.4414 

5018.3887 

10 

Y 

8 

1 

N 

5425,5361 

5019,2129 



subject was allowed to change his expression and move about but was instructed 
to stay within the camera window and focus and to display only frontal views. 

Refer to Table 2. The correlation peaks (XHI) associated with scenes 
AA1-AA5 are very tightly grouped about a value of 5089. All remaining scenes 
are below this value. A threshold of 5089 would generate a simple discrimi- 
nant function capable of separating a correct identification (i.e. , a match 
of person AAl with any of his other facial scenes) from the remaining scenes 
with complete accuracy over the test data. See Fig. 2-12. Similar results 
are obtained using (1) reduced word size and (2) reduced data matrix. In 
all cases it was possible to obtain a linear discriminant function that would 
give 100 percent separation over the test data. 

In order to evaluate the system performance a performance index (PI) was 
defined as 

min over w,N . (T ,T ) - max over w„,N ^(T ,T ) 

1 x’ y^ 2’ g t' x’ y' 

7 .T, rp N X 1000 

min over w, ,Ng^^(T^,Ty) 

This index measures a normalized distance between the clustered correct 
correlation peaks and the closest mismatched peak. 

The effect of varying the bandT-ridth (lines /aver age) and word size is 
shown in Table 3. Three significant aspects are immediately apparent: 

1, For large word sizes, the PI peaks at 1 ines/ average .= 2 and 
then decreases markedly. Specifying two lines/average reduced 
the scene to 16 x 64 data pairs and allows a vertical spatial 
frequency resolution of 32 harmonics. It was pointed out pre- 
viously that the power spectrum leveled off after the 32nd har- 
monic and thus agrees well with the concept of Information 
bandwidth. 

2. The PI is fairly constant over the values of 1 to 4 for lines/ 
average. While a peak is obtained at 2, using half again as 
much data only slightly reduces the PI, Accurate results could 
thus be obtained with reduced system complexity and increased 
throughput . 
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5060.0 5070.0 5080.0 5089.0 5089.5 


Template: AAl 

X = Scenes AA1-AA5 
0 = Other Scenes 


Figure 2-12. A Linear Discriminant 



TABLE 3 

Word Size versus Bandwidth 


Lines/Average 


Bits 

1 

2 

4 

8 

16 

32 

64 

10 

1.927 

1.940 

1.850 

1.502 

1.196 

.6726 

.3623 

9 

1.939 

1.944 

1.853 

1.505 

1.1976 

.6738 

.3634 

8 

1.950 

1.953 

1.860 

1.517 

1.201 

.8172 

.279 

7 

1.964 

1.9692 

1.876 

1.526 

1.217 

.6871 

.184 

6 

2.001 

2.004 

1.918 

1.573 

1.250 

.7129 

.3833 

5 

2.048 

2.070 

1,983 

1.639 

1.304 

.7522 

.313 

4 

2.278 

2,308 

2.212 

1.905 

1.427 

.8402 

.365 

3 

4.300 

4.283 

4.003 

3.427 

2.719 

1.731 

1.559 

2 

13.88 

13.816 

13.43 

10.47 

7.88 

4.703 

1.939 

1 

47.968 

47.502 

47.152 

38.14 

30.167 

15.019 

5.23 
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3. A large increase in the index results as the word size is 

reduced. This effect has immediate cost benefits: 

a. Since excellent recognition can be achieved with only one 
bit, an inexpensive high-speed comparator can be substi- 
tuted for a slower, more costly, 10-bit A/D. Also, the 
noise benefits of large quantization levels are realized 
and thus loosen many potentially expensive noise reduction 
criteria. 

b. In terms of storage a typical 16-bit word could 
hold two lines of video at eight samples/line, 

1— bit samples. An entire condensed array would fit into 
256 words (128 x 16 data pairs (complex data) = 4048 bits = 

256 16— bit words.) The cost reduction is obvious. 

c. The Boolean algebra associated with single-bit multiplica- 
tion and summation can be easily and quickly handled with 
hardware devices. It is conceivable that the entire pro- 
cess could be carried out in near real-time circumstances, 
with a 16-bit mini- or high-speed 8-bit microcomputer con- 
taining only limited core. 

The above three points are worth repeating. They point the way to an inexpen- 
sive, fast, and highly reliable pattern recognition system that can be fabri- 
cated xnth existing technology. Using charge-coupled devices would even fur- 
ther increase the system speed, although at an unknown cost index (measured 
against existing TTL devices) and availability. 

Filtering 

Another aspect worth mentioning is that of high frequency variance. Con- 
sider the power spectrums pictured in Fig. 2-13. They appear remarkably simi- 
lar, expecially at the high-power, low-frequency terms. This is intuitively 
appealing, since faces, after all, are quite simiilar, (General shape, tx 70 
eyes, a nose, mouth, etc., are common features.) Apparently it is small, 
high spacial frequencies which contribute significantly to the differences. 

To test this theory a high-pass filter was implemented (Fig. 2-8 and 
2-9.) The result of this filter on various word sizes for eight lines/average 
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Figure 2-13. Power Spectra 


is tabulated in Table 4. Apparently, facial scene differences are contained 
in the upper spatial fretjuencies . Analog preprocessing of the incoming 
video signal could thus significantly improve the system's performance index. 

It should be pointed out, however, that, if high frequencies are to be 
retained, the sampling frequency must remain high and the data matrix must 
remain large enough to resolve the higher order components. The price of 
increasing the PI via this technique is thus paid for with higher speed 
devices and larger memory requirements. 

CCD Implementation 

A proposed facial recognition system employing charge-coupled devices 
to perform transform operations is outlined in Fig. 2-14 [13], Note that 
the data presented to the camera now becomes the template and not the stored 
scene. 

If a pattern Gj^(f^,fy) is presented and results in a ''0" or mismatched 
output, the scene pointer is indexed to the K+1 scene and the process re- 
peated. Any successful match (a "1" output) stops the process, and access 
(in a security system, for example) is allowed. Access is denied if no match 
is achieved. 


Recognition of facial scenes can be achieved with high accuracy using 
normalised "cross correlation via transforms. Detailed investigation of 
spatial frequencies reveals that accurate identification can be achieved 
using low sampling rates at reduced bandwidths and small array sizes. Per- 
formance can be improved considerably using coarsely quantized gray levels 
and preprocessed data, although the later imposes some restrictions. 

Such "brute-force" technique for pattern recognition may now find 
practical applications because of new CCD devices like the Chirp Z— trans- 
form circuit. Although the system described above could be implemented 
using all digital technology, the overall advantages of a CCD system make 
it far more attractive. 



TABLE 4 


Various Word Sizes for Eight Lines/Average 


Word Size (Bits) 

10 

9 

8 

7 

6 

5 

4 

3 

2 

1 


Performance Index 

58.29 
58.53 
58.67 
58.05 
58.57 

59.30 
72.10 
96.91 

243.00 

513.3 
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III. A PROGEAMM&BLE DISCRETE ANALOG PROCESSOR FOR PATTERN RECOGNITION 

This project in the area of discrete analog processing involves the 
design and implementation of a high-speed unit capable of performing numerous 
transversal filtering operations on a fixed record of analog input samples. 
Such a system is necessary because CCD filters are not yet commercially 
available. The design has been executed using available off-the-shelf logic 
and discrete analog components to attain speed and flexibility of usage. At 
the same time, some sacrifices in package count and circuit integrablllty 
have been made. The unit under construction is thus not a prototype but a 
tool for studying the properties and requirements of a class of similar 
systems which will employ CCDs. In addition, it also allows investigation 
of Interface requirements for microcomputer control of CCD-type processors. 

The main subsystems are a programmable sequential filter (PSF) and a 
signal comparator. Of these, the former is far more complex and has re- 
ceived exclusive attention thus far. Both subsystems run as slaves to a 
microprocessor-based controller which, during operation, performs overhead 
functions periodically and, during test and development, provides a flexible 
man/machine interface. 

The functional organization of the PSF subsystem is shown in Fig, 3-1. 

The microcomputer controls the PSF via an 8-bit status/ command latch. In 
return it monitors the filter's computational completion status via an inter- 
rupt line. Digital logic regulates the loading (sampling) of an input analog 
signal ^?ith respect to number of samples and possible input source multi- 
plexing, the necessary data being encoded in ROM. The samples are shifted 
into a bucket-brigade analog shift register with broadside readout capabili- 
ty. Other logic controls the subsequent computing phase in which the various 
transversal filter outputs are developed sequentially as analog outputs. That 
is, for an input vector x and a transformation matrix A the components of Ax 
are generated. The structure of the computation (i.e, , dimensions and pos- 
sibly the sparseness of A) is once more ROM-encoded, while the specific 
matrix elements are storable with 8-bit accuracy in a coefficient RAM under 
microcomputer control. 
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To process the input sample vector x its i components are read out of 
the broadside shift register taps through a fast analog multiplexer and 
presented sequentially as reference input to a multiplying D/A converter 
(MDAC), Simultaneously, the filter coefficients are read from RAM, and 
analog product outputs are generated which are stimmed by a resettable in- 
tegrator. Data throughput is enhanced by overlapping the multiplication 
and summation operations. The readout process is repeated until all of the 
required filter outputs have been generated. 

One feature thought to be desirable in a non-dedicated processor is 
automatic ranging of the input signal to the internal signal range most 
compatible with linearity of the analog memory device and maximtim overall 
S/N ratio. The PSF analog input circuitry is capable of shifting and am- 
plifying the input to achieve the desired internal signal range. The offset 
and gain required are stored as signals on high- impedance gates and are 
updated periodically by the response of feedback control loops to inputs 
representing maximum and minimum input signal levels. In addition, the 
PSF is designed to periodically store a declared numerical zero signal level 
on a holding gate in order to provide flexible four-quadrant capability. 

The tasks of controlling the autoranging and numerical-zero-acquisition 
circuitry are the "overhead" tasks previously mentioned which fall to the 
microcomputer. To date, the following steps in development of the PSF 
have been accomplished; 

1. Completed design, construction, and test of mictocomputer and 
random logic control circuitry. 

2. Test and measurement of input/output characteristics of the 
selected analog memory device. 

3. Design, breadboard, and test of auto-ranging control loops and 
sample/hold circuits. 

4. Breadboard of multiplier and resettable integrator circuitry. 

Final assembly on circuit cards of the entire PSF is presently nearing 
completion. Remaining work includes in-circuit testing of the analog multi- 
plexer, multiplier, and integrator and verification of control timing rela- 
tionships and computational functions. 
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When completed, it is intended that this system will serve as a 
purpose OP-SAP emulator. It will be used as a feature extractor and 
classifier xn several different pattern recognitxon system studies. 


gener al- 
as a 
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IV. DISCRETE WORD RECOGNITION 


Introduction 

The potential of CCD technology for perforraing fast and complex proc- 
essing of speech signals has generated interest in the possibility of voice- 
actuated co mman d and control systems for aviation. The system envisioned 
would be capable of recognizing discrete words spoken by an operator for 
whom it had been previously trained. The commands would include such things 
as requests about altitude, range, speed, etc. Described below is an in- 
vestigation involving a theoretical study of the application of certain 
CCD devices to signal processing techniques used in automatic speech recog- 
nition. Discrete word recognition systems generally use the same approach 
as any other pattern recognition system. This process includes the 
measurement of the input pattern, the creation of features from these 
measurements, and the identification of the pattern using the features. 

A slight shift in emphasis, however, may occur in the implementation of 
the two systems. In general, the major area of interest in discrete word 
recognition systems is the feature extraction process while using a relative- 
ly simple classification scheme. This may or may not be the case for a 
t]?pical pattern recognition system. Since the classification scheme in a 
word recognition system is simple, a good feature set is imperative. 

Schafer and Rabiner [1] report that both time domain and frequency 
domain feature sets can be used for speech recognition. Popular time 
domain -methods , however, still result in a spectral estimate of the speech 
waveform. Ichikawa [2] did a study on various parameter sets for speech 
recognition. Fig. 4-1 shows the relationship of these parameter sets. 
Ichikawa determined that of the four different parameter sets which he 
tested, the cepstral envelope technique yielded the best results. 

A familiar method for feature extraction from speech data is to 
employ multiple bandpass filters to generate a rough spectral envelope. 

Hatan [3] describes a speech recognition system for a numerical control 
language. Feature extraction was performed by 24 bandpass filters over a 
range of 110-7000 Hz. A recognition rate of 98% was achieved on 26 word 
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Figure 4~1. Relationship of Various Parameter Sets 
for Speech Recognition [6:22]. 



vocabulary having a well-defined syntactical structure. This system also 
boasted real time operation capability. 

It was concluded from reports such as described above that the spectral 
envelope of a speech waveform was generally required in the feature extrac- 
tion process. The spectral envelope may be used either exclusively as a 
parameter set, as a part of a parameter set, or to derive a parameter set. 

It was, therefore, decided to investigate the application of the Fourier 
transform CCD processor to speech waveforms. The actual device of interest 
is the power spectrum processor which uses the sliding Chirp z-transform 
algorithm. The possible availability of such a device in the near future 
was a large factor in the choice of the methodology [4] . 

The investigation was divided into two phases. The first phase involved 
the computer simulation of the CCD power spectrum processor. The current 
lack of availability of the device made the simulation necessary. Since 
the CCD device implements the chirp z-transform with a sliding input data 
window. It was also necessary to generate the power spectrum via a fast 
Fourier transform routine to verify the results. The second phase included 
processing actual speech signals by the simulation routines. A routine used 
to determine the spectral content of an utterance from short time segments 
was also simulated using CCD technology and tried on actual speech data. 

CCD Power Spectrum Device Simulation 

The charge-coupled device implementing either a discrete Fourier 
transform or a power spectrum uses the Chirp z-transform algorithm [5]. 

The discrete Fourier transform is defined as 


N-1 




n=0 


X e 
n 


- 2 irink/N 


(4-1) 


By making the substitution 


2nk = k^ + n^ - (k - n)^ 


(4-2) 


the discrete Fourier transform becomes 
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_ -iirk^/N I -nrn^/N] x7r(k-n)^/N 
n=0 \ 


(4-3) 


This equation is known as the Chxrp z-transform (CZT) equation for the 
discrete Fourier transform. The CZT consists of three basic operations 
in the following sequence: 

1) the multiplication of a complex input x^ by a complex chirp 
waveform, 

2) the convolution of this premultiplication and a complex chirp 
waveform, and 


3) the multiplication of the convolution result and another complex 
chirp waveform. 

When the power spectrum is desired the postmultiplication is not required. 
The equation for the power spectrum is 




(4-4) 


For the case of real input signals, (4) can be expanded to become 


2 (x cos rn^/N) cos ir (k-n)^/N 
n=0 ^ 


+ y (x sin Trn^/N)sin •ir(k-n)^/N ^ 
n tL 
n=0 


+ ^ (x cos rn^/N) sin TT(k-u)^/N 

n=0 “ 


- y (x sin Tm^/N)cos (k-n)^/N ^ 
n O’ 
n=0 


(4-5) 


The power spectrum equation is implemented using multiplying digital-to- 
analog converters for the premultiplication and transversal filters for 
the convolution having impulse responses of 
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= sin. TTm^/N , m = 0, 1, 2, H - 1 


(4-6) 


and 


h = cos irm^/N , m = 0, 1, 2, ..., K - 1 (4-7) 

IQ. \ 

An N-point power spectrxm device would require four transversal filters, 
each 2N stages long. Also a true CCD power spectrum processor would 
require hardware to accomplish 50% blanking of the input. These restric- 
tions led to the implementation of the sliding CZT for spectral analysis 
in CCD devices. The sliding CZT is defined as 

s r -2irink/N ,, 

n=0 

The sliding CZT equation expands as does the true CZT. Each Fourier 
coefficient, however, is now computed over a slightly different set of 
input data points. The advantage of using the CZT is that the filters can 
be N stages long for an N point spectrum and no blanking is necessary. The 
problem lies in the fact that if the input data is not periodic or stationary, 
the sliding CZT will not yield the exact spectral coefficients as does the 
true CZT implementation. A computer simulation package was needed to 
determine if this restriction would greatly hamper the use of the sliding 
CZT on speech waveforms. 

The simulation package consists of two major routines. The first 
routines calculates the power spectrum of N data points by a fast Fourier 
transform algorithm. The other routine calculates the power spectrum 
using the sliding CZT algorithm. Both routines access input data from 
digital magnetic tape. The data is stored in single data item records 
in order to be able to change the spectrum resolution easily. Both 
routines also contain graphic capabilities for ease of analysis. 

The sliding CZT simulation program implements Eq, 4-5. The convolution 
operation is performed in a subroutine with the premultiplication and 
sliding data buffer operations being done in the main routine. Data windows 
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can also be applied to the input data; however, results are only included 
for rectangular data windows. The CCD simulation routine only includes 
the mathematics of the device and none of the electrical properties such 
as transfer ineffxcxency. 

The software packages also includes a data acquisition routine. The 
routine samples the input at prescribed intervals and for a specified 
duration and stores them on magnetxc tape in a format compatible with the 
sxmulatxon routines. This extra storage medium is necessary for ease of 
analysxs and because the simulation routines are not capable of real txme 
operation. Listings of the simulation routines are found in the Appendix. 

Speech Waveform Analysis Using Simulation Package 

The real time speech data was acquired in the Computer Systems Labora- 
tory by a Hewlett Packard 2100A computer system. The simulation package 
was also resident on this computer system. The computer system contains 
a general purpose data acquisition system capable of acquiring data samples 
at any rate up to 200 KHz. A block diagram of the system is shown in 
Fig. 4-2. The sampling process can be initiated by either the computer 
alone or a combination of the computer and an external syncronization 
signal. The system also contains selectable low-pass filters to remove 
alaising problems. Special circuitry was necessary to acquire speech data 
using the data acquisition system. A block diagram of the speech interface 
is shown in Fig. 4-3 [6], A portion of the interface was devoted to the 
generation of the sync signal. The speech waveform was passed through an 
automatic gain control (AGC) circuit to force the signal to occupy the 
full dynamic range of the A/D converter. Rectification circuitry was also 
available although it was not used for this investigation. 

It is fairly widely accepted that systems for speech recognition can 
be band-limited to around 4 KHz, thereby requiring at least an 8 KHz, 
sampling rate. It was also determined that a one-half second interval is 
sufficient for most utterances of initial interest. These requirements 
then dictate the -processing of approximately 4096 samples per utterance. 
Because of the large number of samples per utterance, the power spectrum 
of an entire record was impractical to obtain. A recognized way of 
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Figure 4-2. Data Acquisition System 
Block Program [6;43] 
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accomodatitig thxs amount of data Is to break th.e sampling interval into 
sub-intervals. The size of each sub-interval is a function of the frequency 
resolution necessary for correct classification. The power spectrum of 
each sub- interval is then obtained. This power spectrum is known as a 
perlodogram. A smaller number of perlodograms per utterance can be 
generated by the averaging of adjacent perlodograms. These resulting 
perlodograms are then used in most speech recognition systems to create 
the feature set. 

Several one-half second utterances were input to the simulation 
package. The 4096 data samples were divided into sub-intervals of 128 
samples each. This sample size produced a frequency resolution of approx- 
imately 62 Hz. Ten adjacent perlodograms were then averaged to form a new 
perxodogram in order to reduce the total number of perlodograms in the entire 

utterance. Figs. 4-4 and 4-5 show the results of the first averaged peridogram 
of the utterance, "Alpha”. Fig. 4-4 shows the true power spectrum including 
the negative frequency terms. Fig. 4-5 shows the result of the CCD simulation 
with its negative frequency coefficients. The negative frequency coeffi- 
cients were included for comparison since the sliding CZT algorithm does 
not gttarantee that these coefficients are merely a reflection of the 
positive coefficients. Figs. 4-6 and 4-7 are similar results of a different 
utterance. 

Conclusions 

The results of the simulation show that CCD power spectrum device 
may find a place in the field of speech recognition. The processing of 
perlodogram averaging tends to cure a major pitfall of the sliding Chirp 
z-transform algorithm. The sliding CZT processor produces a power spectrum 
that is very similar to the true power spectrum for segments of speech. 

CCD devices may possibly rekindle interest in various techniques in speech 
recognition that were previously put aside for various reasons. 
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Fxgure 4-7, Power Spectrum via CZT for the First 
Averaged Periodogratn of "Bravo" 



APPENDIX IV-A 

Charge- Coup led Device Simulation Program 


50 



oo o o o o o 


FTO4 

PROGRAM CCD 

BIT 15 ON FOR INDIVIDUAL ,OFF FOR COl^POSITE 
BIT 0 ON FOR EACH ITERATION RESULT 
BIT 1 ON FOR LABEL 

DIMENSION SINl (51 2), COS 1 (51 2), DATA! (5 1 2) ,DATA2X51 2 ) 
DIMENSION TABLC512) ,C?^POS(51 2) 

DIMENSION JUNK(10) 

DIMENSION NAME (40) 

COMMON ISl ,IS2,IS3,IS4,IS5,IS6,R1 , R2 , R3, R'l , IS7 , IS8 ,P5 , R6 
INTEGER A 
REAL ISPY 
WRITEd ,500) 

500 FORMATC'INPUT TYPE OF PLOT - 1 . ,0 , 1 " ) 

READ( 1 ,-<^) ITYP 

. WRITEd ,501 ) 

501 FORMAT(" INPUT HI Y") 

READ( 1 ,*) HIY 

WRITE( 1 ,502) 

502 FORMAT(»TYPE 1 FOR LOG, 0 FOR LIN") 

READd,*) ILOG 

WRITEd ,2) 

2 FORMATC'TYPE CCD WINDOW SIZE '* ) 

READd ,*)N 

WRITEd ,400) 

400 FORMATC'TYPE # WINDOWS FOR COMPOSITE”) 

READ( 1 ,*)NW 
CALL WINDO(0, 1 ) 

WRIlEd ,3) 

3 FORMAT (“TYPE TAPE FILE NUMBER") 

READ( 1 ,*) lOP 

CALL PTAPE(8,IOP,0) 

WRITEd ,13) 

13 FOR?MT("TYPE ONE IF HEADER ON TAPE, ELSE 0") 

READ( 1 ,*) IHD 

IFdHD. EQ.0) GO TO 14 
DO 6 1=1 ,9 

READ(8,7) (JUNK( J) ,J=1 ,10) 

7 FORMAT(10A2) 

6 CONTINUE 

READ(8,8) (JUNK( J) , J=1 ,10) 

8 FORMAT(10I6) 

14 CALL CONV(N,COS1 , SINl ) 

CO 10 1=1 ,N 
READ(8)A 

DATAl (N-m )=A*COS1 (I)/N 
10 DATA2(N-I + 1 )=A*SIN1 (I )/N 

DO 9 L=1 ,N 
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9 CMPOS(L)=0. 

LC=0 

1 ] DO 1 20 L=1 ,NW 

LC=LC+1 
DO 100 1=1 ,N 
DO ‘/O I2=1,K'-1 
IQ=N-I2 

DATA! ( 10+ 1 )=DATAI (IQ) 

70 DA TA2 ( I Q + ] ) =DATA2-( 1 0 ) 

READ(8)A 

DATA1 (1 )=A+C0S1 (I)/N 
DATA2(1 )=A+SIN1 (I)/N 

CALL C0NV(N,C0S1 ,SIN1 ,DATA1 ,DATA2,RESL) 

11=1-1 

TABLC I )=RESL 

CMPOS ( I ) =CMPOS ( I) + ( RESL-CyPOS CD) /LC 

100 CONTINUE 

IF( ISSW(0)) 102, 101 

101 IF(L.NE.NW)GO TO 120 

102 Bl=N-l 

CALL CHRS(27, 12,128) 

CALL RANGE(0. ,B1 ,0. ,HIY,0, ILOG) 

CALL AXES (0. ,0. ) 

ISPY=Bl/]0 

CALL TICK(0,0. ,B1 ,ISPY,30, 1 ) 

ISPY=HIY/10 

CALL TICKd ,0. ,HJY, ISPY,30, 1 ) 

IF( ISSl'IX 1 5) ) 1 15, 116 

115 CML GRAPH(nyP,N,TABL) 

GOTO 1 1 8 

116 CALL GRAPH( HYP, N, CMPOS) 

118 IF (ISSW( 1 ) ) 119 , 1 20 

119 WRITE(6,200) 

200 FORMATC "SET CURSOR, STRIKE A KEY, AND INPUT LABEL:") 
CALL CURS KIND , IXPOS,IYPOS) 

CALL TPLOTC0, IXPOS, lYPOS) 

CALL CHRS(31 ,128,128) 

READ(6,210) (NAME(I) ,1 = 1 ,40) 

210 FOR?AT(40A1 ) 

WRITE (1 ,210) (NAME(I) , 1=1,40) 

WRITE (6, 220) 

220 FOR MAT ("TYPE 1 FOR MORE LABELS, ELSE 0") 

READ(6,*)IHD 
IFdHD.EQ. 1 ) GO TO 119 

120 CONTINUE 
WRI1E(6, 1 2) 

12 FOR MAT( "MORE DATA... 1 FOR YES ELSE 0 ") 

READ(6,*)ILP 
IF (ILP.EQ. 1 )G0T011 
END 
C 
C 
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SUBROUTINE CONV(N,COS1 ,SIN1 ,DATA1 ,DATA2,RESL) 
DIMENSION COS 1(1) ,SIN1 ( 1 ) ,DA7A1 ( 1 ) ,DATA2( 1 ) 
DATAK/0/ 

K=K-il 

IF(K.GT. 1 )GOTO 50 

no 10 1=1 ,N 

11 = 1-1 

COS1 (I)=C0S(3.14159*I1*I1/N) 

10 SIN1 ( I)=SIN(3. 141 59*1 1 *11 /N) 

RETURN 
50 S1=0. 

C1=0. 

S2=0. 

C2=0. 

CO 70 1 = 1 ,N 

C1=C1 -0ATA1 (I )*COSl (I) 

S1=S1+DA.TA1 (I)*SIN1 (I) 

S2=S24DATA2(I)*SIN1 (I) 

70 C2=C2+DATA2(I)*C0S1 (I ) 

RESL=(S2HC1 )*(S2+CI ) ^ (S I -C2 ) *( S 1 -C2 ) 

RETURN 

END 

END$ 
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APPENDIX IV-B 


Fast Fourier Transform Power Spectrtan Program 
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ooo oooooo ooooo 


FTN4 


PROGRA?^ CCFT 


THIS PROGRAM COMPUTES THE POWER SPECTRUM OF N DIMENSIONAL 
RECORDS OF DATA. A COMPOSITE SPECTRUM CAN ALSO BE GENERATED. 

THIS PROGRAM IS THE ANALOG OF CCD PROGRA?/. 


BIT 15 ON FOR INDIVIDUAL ,OFF FOR COMPOSITE 
BIT 0 ON FOR EACH ITERATION RESULT 
BIT i ON FOR LABEL 


DIMENSION ARRY(2,512) ,TABL(512) ,CMP0SC5! 2) 

DIMENSION JUNKCl 0) ,MAME(40) 

COMMON IS1 ,IS2 JS3,IS4,IS5,IS6.R1 ,R2,R3,R4, IS7, IS8,R5,R6 
INTEGER A 
REAL I SPY 
WRITEd ,500) 

500 FORMATC INPUT TYPE OF PLOT ~1.,0,1‘') 

READ( 1 ,*) ITYP 

HRITE< 1,501) 

501 FORMATC' INPUT HI Y" ) 

READCl ,*) HIY 

WR ITE( 1,502) 

502 FOR MAT(“ TYPE 1 FOR LOG, 0 FOR LIN") 

READd,*) ILOG 

WRITEd ,2) 

2 FORMAT ("TYPE FFT WINDOW SIZE") 

READ( 1 ,*)N 

WRITEd ,400) 

400 FORMATC'TYPE # WINDOWS FOR COMPOSITE") 

READd ,*)NW 
CALL WINDOC0, 1 ) 

WRITEd ,3) 

3 FORMATC'TYPE TAPE FILE NUMBER") 

READ( 1 ,^)IOP 

CALL PTAPE(8,IOP,0) 

WR I TE d , 1 3 ) 

13 FORMATC'TYPE ONE IF HEADER ON TAPE, ELSE 0") 

READd ,*)IHD 

IF(IHD.EQ.0) GO TO 14 
DO 6 1=1,9 

READ(8, 7) (JUNK(J) ,J=i ,10) 

7 FORMAT(10A2) 

0 CONTINUE 

READ (8, 8) (JUNK(J) , J=1 , 1 0) 

8 FORMAT(10I6) 

14 DO 9 L=1 ,N 
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9 CMPOS(L)=0. 

LC=0 

n DO 1 20 L=1 ,NW 

LC=LC +i 
DO 50 1=1 
READ(8)A 

ARRY( 1 , I)=FLOAT(A)/N 
50 ARRY(2, I)=0.0 

CALL FFT(ARRY,N,-1 ) 

DO 55 1 = 1 ,N 

TABL(I)=ARRY( I ,I)MRRY(1 , 1 ) +ARRY( 2 , 1 ) *ARRY C 2, 1 ) 

55 CMPOSd )=CMPOS{I) + (TABL(I)-C//POS( I ))/LC 

IF(ISSW(0))102, 101 

101 IF(L.NE.NW)GO TO 120 

102 B1=N-1 

CALL CHRS(27, 1 2,1 28) 

CALL RANGE(0. ,B1 ,0. ,HIY,0, ILOG) 

CALL AXES (0. ,0. ) 

ISPY=B1/10 

CALL TICK(0,0.,B1 ,ISPY,30,1 ) 

ISPY=HIY/10 

CALL TICKC 1 ,0. ,HIY, ISPY,30, 1) 

IF(ISSWC15))1 15,116 

115 CALL GRAPH (ITYP,M,TABL) 

GOTO 118 

116 CALL GRAPH(ITYP,N,CMPOS) 

1 18 IF (ISSi\( 1 )) 1 19 , 120 

119 WRITE(6,200) 

200 FORMATC'SET CURSOR, STRIKE A KEY, AND INPUT LABEL:") 
CALL CURSKIHD , IXPOS,IYPOS) 

CALL TPLOT(0,IXPOS,IYPOS) 

CALL CHRS(31 , 128,128) 

READ(6,210)(NAf^E(I),I = l ,40) 

210 FORMAT(40A1 ) 

MRITE( 1 ,210) (NA?^E(I ), 1=1 ,40) 

WRITE (6, 220) 

220 FOR ?/AT( "TYPE 1 FOR flORE LABELS, ELSE 0") 
READ(6,*)IHD 
IFCIHD.EQ.l ) GO TO 119 

120 CONTINUE 
WRITE (6, 1 2) 

12 FORMAT("?AORE DATA... 1 FOR YES ELSE 0 ") 

READ(6,*) ILP 
IF (ILP.EQ. 1 )G0T01 1 
END 
C 
C 

SUBROUTINE FFTCDATA ,NN , IS IGN) 

DIMENSION DATA(l) 

SIGN=FLOAT(ISIGN) 

N=2^NN 

J=1 
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Co IV) 


DO 5 1=1 ,N,2 
IF( I-J) 1 ,2,2 
TE?v1FR=DATA(J) 

TFMFI=DATA(J+! ) 

DATA( J)=DATA(I ) 

DATA(J+1 )=DATA(H1 ) 

DATAC I)=TEMPR 
DATA( 1+1 )=TEMPI 
M=N/2 

IF(J-M)5,5,4 
J=J-M 
U=U/2 

IF(M-2) 5,3,3 

5 J=J+M 

¥mx=:2 

6 IF(MMAX-N)7,9,9 

7 ISTEP=2+MMAX 
ZAM=FLOAT(M?/AX) 

DO 8 M= 1 , HMAX , 2 

THETA=3 . U 1 592 +FLO ATX M- 1 ) /2A¥ 
1\(R=COS( THETA) 

ARG=1 .-WR+‘**!R 
IF (/iRG) 10,11,11 

1 0 ARG=0 . 

11 1Vl = SIGN*SQRT(ARG) 

DO 8 I=M,N,ISTEP 
J=I+MMAX 

^EMPR=WR+DATA(J)-WI*DATA{J^1 ) 
TEMPI=WR*DATA(J+1 ) + WI+DATA( J) 
DA TA ( J ) =DATA ( I ) -TEM PR 
DATAC J+1 )=DATA(I + 1 ) -TEMPI 
DATAC I )=DATAC I ) 4TEMPR 

8 DATACI+1 )=DATACr+1 )+TEMPI 
MMAX=ISTEP 

GO 10 6 

9 RETURN 
END 
END$ 
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V. RESOLUTION ENHANCEMENT 


Introduction 

A novel application of charge-coupled devices (CCDs) for data pro- 
cessing to a unxque electro-optxcal system has been studxed and is presented 
xn this part of the report. The electro-optical system employs a tech- 
nxque for binary monochromatic resolutxon enhancement beyond the capability 
of an indivxdual sensor. The system utilizes a gray-level to binary-level 
conversion and sensor motion to extrapolate data values smaller than the 
individual sensor area resolution capability. 

Such a system may find practxcal applxcation in such dxverse fields 
as satellite and aerxal reconnaissance to medxcal tomography or any applica- 
tion where greater resolving power is needed than a discrete sensor can 
provide. Since the technique requires only a sensor capable of gray- 
level inf ormtion, any further advances in technology should only serve to 
further the technique's utility. With recent studies of multi-level logic 
systems [1] and wxth CCDs inherent capabxlity for multi-level output, a 
suggested projection would be a combinatxon of sensor and pfocessxng system 
whxch could contribute to a "smart sensor." 

Pursuing such a promxsing technique into realxzatxon, the implemen- 
tation, however, poxses some difficulty due to the characteristically 
large amounts of data generated. This data, when processed by McRee’s 
technique [2], becomes bulky. A paper by Henderson and McVey [3], dis- 
cussed later, proposes a more sophisticated algorithm whxch alleviates 
some of the conditions imposed by McRee at the expense of even more data 
manipulation and storage. A mode of reducing such manipulation or immense 
storage, yet not at the expense of the improved Henderson technique's bene- 
fits, is deemed necessary. Such a mode of reducing the manipulations or 
storage might exist in the form of a filtration process occurring at some 
location in the system. With data available in either sequential or block 
information, particularly if a square matrix or picture sensory array is 
used, a two-dimensional filter or a high-speed sequential filter would be 
indicated. With high speed, low power/density dissipation, and analog and 
digital processing, CCDs appear to be ideal for this application. This 
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IS reinforced by their flexibility in assuming the form of either a two- 
dimensional or high-speed sequential filter. 

This application shall be approached in this report by first pre- 
senting and exploring McRee’s technique [2] into the detailed phase of an 
example. Next, Henderson’s algorithm [3] shall be presented in a similar 
manner with the last part of the presentation being devoted to the prac- 
tical application of CCD technology. The practical application phase is 
the major topic presented and will be pursued in depth as various segments 
of the research are developed. 

The presentation of the developed research shall progress through five 
phases. Phase one introduces previous research effort by McRee, et al. In 
phase two, several possible CCD applications for the enhanced resolution 
system are developed with phase three consisting of the phase two applica- 
tions presented as examples to illustrate the limitations of each applica- 
tion. Sources of error in the system algorithm and in the device implemen- 
tation will be discussed in the fourth phase. The last phase concludes the 
report with a summary, recommendations, and conclusions based on the illus- 
trative examples. 

In the method for enhancing resolution in electro-optical systems with 
a multi-level output that has been presented by McRee and McVey [2], the 
resolution enhancement entails special logic circuits which allow discrimi- 
nation of events smaller than the area of the discrete sensor. In a spatial 
transformation of numeric systems, gray-level information of a fixed resolu- 
tion is mapped onto a binary-level system with a resulting resolution greater 
than that of the gray-level system. McRee has formulated a linear transfor- 
mation equation 

S = MT (5-1) 

for this conversion in addition to a technique for its realization, where 
S is defined as a vector composed of the elements s^^ which are the sensor's 
output with r as row and c as colximn: M is defined as a p x q transformation 
matrix with m. . elements and p < q iriLth p sensors and q outputs; and T is 
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where x xs row and j is 


defined as a vector composed of the elements t^^. , 
coltimn xn the target area. 

McRee’s technxque typically utilizes a solid-state, electro-optxcal 
sensor array with an individual sensor area A and sxde length d, where A 
= d^. Refer to Fig, 5—1. In the examples, a five discrete level output 
sensor wxth a resolution xmprovement of 4:1 is assumed. 

Positioning a gridwork over a target area and establishxng a direction 
of travel, as depicted xn Fig. 5-2, the array is fxrst positxoned as in 
Fxg. 5-3. Samples of the target are processed, whereby the amount of light 
energy incident on the sensor is integrated; and this integrated input is 
passed to the specxal processing logxc circuxtry. The processed output, 
if exceedxng 50% of the sensor's maxxmum output, is converted to a 1 state, 
if otherwxse, to a 0 state, i.e., 



This value is stored, the sensor moved to the second position (see 
Fig, 5-3) , and a sxgned algebraxc additxon xs performed between this data 
and the prevxously stored value. This restilt is stored and the process 
contxnued untxl the target has been completely sampled. The matrix of 
binary data thus formed is the enhanced resolution output. 

McRee found that the enhanced resolution matrix was p x q, where p 
< q thus incurring a non-square matrix and the resultant inversion diffi- 
culties. To overcome this imposition McRee established a boundary condi- 
tion (refer to Fig, 5-4) , where 



which reduces his M matrix to a square and invertable matrix. 

The only other apparent restriction is that there is a tendency for 
error to accumulate and propagate as is illustrated in the following theory 
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< d 


Sensor Area = 


Figure 5—1. Sensor Area of Individual Sensor on Array 
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and example. These restrictions are overcome by Henderson’s application 
of estimation techniques to the data which shall be illustrated. 

McRee defines the process of transforming gray information to enhanced 
binary information as image delimition.' As is obvious from Fig. 5-3, the 
enhanced binary resolution is obtained from sensory overlap and the result- 
ant increase in image threshold determination accuracy. This accuracy 
appears from the signed algebraic addition performed by the special logic 
circuits to ascertain the exact location of the sensor loci which, in effect, 
increases resolution. 

A perfect sensor is the first assumption by McRee in his restrictions 
upon the technique justified by several techniques for the elimination of 
dead space between sensors. That sensor area does not approach zero is 
another assimiption which is based on practicality. The last two assumptions 
cause the most difficulty, as shall be discussed. The resultant non-square 
matrix problem is overcome by establishing boundary conditions; and, by 
assuming an error-free or noiseless transformation commencing at the boun- 
dary, McRee introduces a major source of error. 

For the transformation matrix M and its subsequent unit vector columns 

m , let there exist a coefficient 1 such that 
3 2 

l^m^ + l^m^ + • • • + l^m^ = 0 for j = (1, 2, 3, • • • , q) (5-4) 


as a test for linear dependence and has a set 1^ = {-1, 0, +1}, However, 

McRee sets any value 1. < 0 equal to zero, thereby reducing the set to 1. 

“ {0, 1}. Then, the linear transformation equation may be represented by 
an example. 

Arranging a sensor array, as in McRee 's paper and in Fig. 5-5, it 
must be pointed out that there must exist a complete overlap, either through 
exaggerated sensor motion or staggered sensors, for the results to be meaning- 
ful. For an example target image, as in Fig. 5-6, the following manipulations 
are performed from (1), S = MT, 


S 

rc 


(t) 




t 


(5-5) 
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Figure 5-5 , Staggered Sensor Array Shown Positioned 
Over Target Area 







setting t as time for sensor positions, as in Fig. 5-7, and n as the number 
of columns in the sensor to be processed, it is seen that p = nt + (n - 
l)t = (2n - l)t and q = 2nt for the transformation matrix M. The equation 
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results. However, if boundary conditions, as in (5-4), are implemented, 
the equation above reduces to that of (5-7), where p = q = (2n - l)t, as 
illustrated in Fig. 5-8 and 


69 








Direction of Travel 




Boundary 
' Conditions 



Sample 

Target 



Staggered Sensor Array 


Target Area with Sample Target 


Figure 5-8. Staggered Sensor Array with Sample Target Illustrating 
Boundary Conditions 






Sii(l) 

- 3 


1 

0 

0 

0 

0 

0 

0 

0 

0 


” “ 

^22 


- 2 


1 

1 

0 

0 

0 

0 

0 

0 

0 


^23 

SjjC3) 

- 2 


0 

1 

1 

0 

0 

0 

0 

0 

0 



Sijtt) 

- 2 

- 

1 

0 

0 

1 

0 

0 

0 

0 

0 


^32 

Si2<2) 



1 

1 

0 

1 

1 

0 

0 

0 

0 


^33 

Si2<3) 



0 

1 

1 

0 

1 

1 

0 

0 

0 


^34 


- 2 


0 

0 

0 

1 

0 

0 

1 

0 

0 


%2 

Sji (2) 



0 

0 

0 

1 

1 

0 

1 

1 

0 


%3 

^21 



0 

0 

0 

0 

1 

1 

0 

1 

1 


%4 


(5-7) 


Inverting and performing the matrix multiplication, the equation 
above can be expressed as its components which are the enhanced binary 
resolution algebraic equations. 


‘" 22 “ 


“Sii(l) - 3 

^23 


Sii(2) - S^^(l) + 1 

^24 


Sii(3) - Sjj(2) + Sj^(l) - 3 

^32 


^12 ^ 

^3 3 

- 

(2) - S.,(l) - f2) + fl) +1 

^34 


- Si 2(2) + Si2(l) - Sii(3) + Sii(2) - Sn(D + 1 

*^42 


- Si2(l) + Sii(l) - 3 

^3 


S^j(2) - S^jCl) - s^^(2) + S^^(l) + Sj^(2) - S^^(l) +1 

t 

44 


S^^(3) - 821 ( 2 ) + 821 ( 1 ) - S 12 O) + Si 2 ( 2 ) - 812 ( 1 ) + Sii(3) - 

_ -J 


811 (2) + S^j(l) - 3 


Then, substituting values for the above 

S (t) = [3 2 2 3 2 2 3 3 4]^ Cl5-9a) 

rc 
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(5-9b) 


T= [000111011]^ 

vectors result concluding the example (refer to Fig. 5-9). 

McRee has made two assumptions whxch give rise to difficulties as they 
are generalized. With the first assumption noted before that gives diffi- 
culty, McRee asstimes a boundary condition which reduces the transformation 
matrix M to a square matrix; however, such a boundary condition interferes 
with the practical implementation. The second troublesome assumption, that 
the transformation is error-free or noiseless, is seen from the algebraic 
equations previously described to propagate error in the recursive technique, 
Henderson proposes and justifies the removal of the imposing bomdary condi- 
tions as well as ridding the model of the propagating error by application 
of estimation techniques to generate a pseudoinverse. The purpose of 
Henderson’s technique was to eliminate the boundary conditions; however, 
a side benefit not discussed was the removal of the propagated and accumu- 
lated error. 

Directing attention to the removal of these restrictions, Henderson 
begins with a presentation of the classical estimation model 

z = Hx + u (5-10) 

where, z = m x 1 observation vector, H = m x n mapping matrix, x = n x 1 
parameter vector, and u = m x 1 vector of random variables or noise. By 
the direct substitution of McKee's notation into the estimation equation 
(5-1) and (5-11) , the only changes are the resultant treatment of the in- 
verse transformation matrix 

S = MT + u (5-11) 

Henderson treats the estimate as noisefree and applies least squares to 
obtain 
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M'^S 


(5-12) 
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where m"^ is the pseudoinverse matrix of M, and T is the estimate vector. 
Utilizing the maximum row rank property of M, the pseudoinverse is given 
hy 


M*' = (5-13) 

T 

where the M matrix is not a complex matrix hy prior definition. From the 
previous target example for McRee's technique, the sensor values are assigned 
(5-9a) and the pseudoinverse computations performed by TSO BASIC on a 
HP2100 time-share system resulting in 

T= [0000000001110011]^. (5-14) 


Several target transformations are illustrated in Figs. 5-10, 5-11, and 
5-12. The M matrix is computed only once, and (5-14) can be summarized 
by (5-15) , where the number of coefficients is determined by the number of 
sensors and the resolution enhancement ratio, where k, the threshold of 
the image delimitation, is typically set equal to .5 and 




i=l 


+ 

m . s 
ij xc 


(t) 


(5-15) 


This equation is implemented as the design equation for the special pro- 
cessing circuitry and is suggestive of the transversal filter design 
equation (5-16) given by [4] 

k 

V, (t) = y h d(t - mw) (5-16) 

d 1 _ m 

m=l 


where w is the tap delay, k is the number of taps, h^ is the weighting 

coefficient, t is the sample interval, d is the input sample voltage, and 

V , (t) is the filtered output voltage, 
d 

The combination of the techniques of McRee and Henderson has progressed 
to the stage of a practical implementation. The implementation from the 
equation (5-15) , as conceived by McRee, involves addition and multiplication 
as well as an analog-to-digital conversion for each sensor as per the flow 
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chart process summary. Fig. 5-13, However, if analog information can be 
processed instead of digital, a great savings in size and complexity could 
be achieved as well as to reduce the error inherent in quantization. In 
the transversal filter equation (5-16) , with its continuous data sampling 
nature, this achievement is apparent. Therefore, the design equation (5-15) 
implemented as a transversal filter would be an obvious realization; and, 
with CCDs, several design realizations of transversal filters [4, 5, 6, and 
7] have been accomplished. 


System Designs 


i 


Transversal filters will be used in the implementation. A general 
form of the transversal filter is shown in Fig. 5-14 and is described by 


k 

V, (t) = y h d(t - mw) (5-16) 

d m 

m=l 


as given previously. This configuration uses a normalized electrode tapped 
a proportional amount across the width corresponding to the ratio [8] 



where h^ is the weighting coefficient, and is the input charge sample 
in the cell. From (5-16), the function d(t - mw) implies a time delay or a 
sample storage. The term transversal filter is applied to this general 
class of device, particularly where a storage array is implied and the data 
shifted into the register is not part of a continuous input. Consequen- 
tially, the classical definition of the use of the filter as a signal 
detector or emphasis device shall not apply. Instead, discrete sensor 
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Figure 5~13» Sxmplxfied Flow Chart of McRee’s Technique 
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Figure 5-14- Typical Weighted Tap Transversal 
Filter Arrangement C4l] . 





values processed xn a matrix format where each sensor output for a sample 
is wexghted by a coeffxcient, then summed wxth the remaxning weighted sensor 
values to fomn an element output, shall be termed a transversal filter or 
just filter. 

To xncrease the flexibility of the transversal filter, adjustable tap 
weights implemented in place of the fixed tap wexghts would realize a pro- 
grammable filter. Several techniques for realizing a programmable trans- 
versal filter have been xmplemented [4, 6, 9, 10] . As mentxoned in Sectxon 
III, none are available as off-the-shelf components. 

Considering the most direct method or a mxnimal desxgn and maximal 
hardware, there are two xmmedxate approaches to a direct implementation 
of whxch one would be the applxcation of one transversal filter to each 
output element as shown xn Fig. 5-15. Such a desxgn would employ a maxxmal 
number of filters; and, as seen from the figures, there are many modes of 
data input and output. Utilizing a maximal filter design would result in 
the highest throughput of data, particularly if parallel input / output ^ 
structures are adopted; however, the expense of such a high data rate xs 
increased cxrcuit density. It becomes apparent from minxmal size limitations 
in CCDs and the consequential trade-offs between accuracy, error, and size 
that the ensuing number of filters would increase with the dimensionality 
of the pseudoinverse M . Since, in the future, there will undoubtably be 
an increase in technological capabxlxty and the demand for higher speed 
processxng shall contxnue, the maximal fxlter design will be used to xntro- 
duce the desxgn examples and discussxon. 

With the processxng of information in an analog form, an immediate 
savxngs xn a conversxon unit (A/D) per sensor, as proposed by McRee, xs 
realized; whereas, xf CCD technology is employed in the sensor and pro- 
cessing circuitry, a direct interface is then possible. A one-to-one 
correspondence between the filter and resolution element xs best fulfilled 
wxth fxxed tap weights in a transversal filter where each coefficient 
represents a column element in a row or the pseudoinverse The trans- 

versal filter, as described by (5-16), then sums the weighted elements 
which are shifted along the weighting delay line until all samples are 
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present in correct order. The output is strobed at the proper instant to 
either load a multiplexor for a serial output which would ease the conver- 
sion to a standard TV transmission pattern or load parallel output buffers. 
A bonus evident in the output circuitry is that no additional clock speed 
IS required in the serial multiplexor scheme, since there is a finite time 
period between successive conversions through the filter, as illustrated 
by the timing diagram in Fig. 5-16. The timing diagram can be expressed by 

. f , 

COtIPLETE SAMPLE CONVERSION TBIE = T = •§- = ■§- (5-19) 

c f f f 

s cs s 

where f is the shift clock frequency for the filter, f is the sensor 
CS s 

sample rate, d is the sensor resolution enhancement factor, and f is 

CO 

the output clock frequency. 

Two input schemes are indicated by the serial shifting nature of the 
transversal filter. One intuitive scheme would be a simple serial shift 
into the transversal filter, as depicted in Fig. 5-17; the other scheme 
would involve a parallel-to-serial conversion with the effect of increasing 
the speed of the data transfer given the higher clock speed of the filter. 
Another scheme in the same vein of parallel loading might be realized if 
the filter is amiable in a density trade-off, as indicated in Fig. 5-18, 
to a loading in the filter area. Thus, having seen the maximal filter 
approach, an attempt at the maximal design and minimal hardware approach 
will be presented as another implementation. 

Such an approach would require a single transversal filter with a 
variable programmable tap weighting, as described by several experimental 
research groups [4, 6, 9, 10]. The same input and output schemes could be 
utilized with the output possessing an additional intermediate analog delay 
line storage to store the sums-of-products terms from each sensor sample 
position until the complete sample is obtained. The pseudoinverse M 
matrix must be loaded into the programmable filter a row at a time. If 
row-by— rov 7 programming is to be the mode of operation, an analog output 
storage is present, and the filter shift register readout is nondestructive 
and can be recirculated, then no additional input circuitry is necessary. 
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Figure 5-16. Timing Diagram Illustrating the Finite Time Available 
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Figure 5-17. Serial Loading of a Transversal Filter 
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Figure 5-10* Parallel Loading of a Transversal Filter 



However, if the fxlter cannot be recirculated or if analog output storage is 
not present, then there must exist an input storage of sufficient size to 
retain a complete set of sensor readings corresponding to 

= n X d (5-20) 

where n is the number of sensors and d is the resolution enhancement factor 
(e.g., 4:1). There still exists the need of storage for the coefficient 
M matrix whose dimensionality has been previously discussed and examples 
presented. It is apparent that a larger amount of discrete storage cells 
would be necessary if signed digital storage were to be used instead of an 
analog value, owing to the greater radix values available. Digital storage 
technology is well established and continues to increase in density; how- 
ever, there are only two basic types of analog storage currently available. 
The first type is evident in the analog delay line and is comparable to 
a dynamic storage unit or to a recirculating shift register. This dynamic 
storage requires a refresh scheme which could pose problems in its nvnnbers, 
particularly, as a large number of sensors are employed; but, if externally 
accessible, there is the alternative of an adaptive filtering scheme. The 
second type is seen as a static type employing a MOSFET structure in conjunc- 
tion with a charge storage cell [4, 9] or as a tap weight where the tap area 
determines the value for a normalized charge which is then merged into the 
data stream in the shift register (Fig. 5-19). If the MOSFET/capacitor 
structure is employed and a reset line is available, then there also exists 
the alternative of an adaptive filter scheme. With storage of either digital 
or analog values, there still is the imposing dimensionality of the pseudo- 
inverse M matrix, as evidenced by a small sensor matrix of 20 x 20 elements 
requiring 144,400 coefficients to be stored. Another example of a 100 x 
100 element sensor matrix requires 9.8 x 10^ coefficients which would serve 
to illustrate the exceeding large storage necessary for the coefficient 
matrix. There appears to be no easy implementation for the resolution en- 
hancement technique; therefore, a further study of the coefficient matrix is 
deemed advisable. 

Indeed, by studying the pseudoinverse matrix, a pattern of values 
appears to emerge and even the number of discrete values in the matrix seems 
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Data Signal 
Stream Output 


Figure 5-19, Example of Simple Analog Storage Using 
a Proportional T^eighted Tap 




limxted. In constructing the transfonration matrix (5-21), a pattern of 
dxagonalized square unit submatrices xs evxdent and such a pattern would 
xntuitxvely gxve rise to a comparable pattern xn the pseudoinverse M 
matrix, A pattern does appear in the form of a mirror image reflected 
about a central axis. Also, after several examples of varxous sensor array 
sizes were checked a consistent rule of thumb for the number of dxscrete 
absolute values for the coefficxents is intuitively seen to be 

V = 2n (5-22) 

where n xs the number of sensors. An extensive study of the pattern dis- 
played xn the pseudoinverse reveals further possible reduction by consid- 
erxng the submatrxees defxned by the boundaries in (5-23) , If each sensor 
array sample is considered to be a major submatrix of n columns where n 
is the nmber of sensors, then, minor submatrices composed of redundant 
elements can be formed. Redundant rows can be coded as to the number of 
xteratxons xn the minor submatrix by the addition of a sufficient number 
of dxgital bxts. Such a technxque could be called mxcrocoding and, to 
maxntain consxstency, shall be so termed. An instant reduction of the 
dxmensionalxty of the pseudoinverse to that of a microcoded coefficient 
array brxngs increased concentration on the pattern in the pseudoinverse. 
There even appears to be a mirror image in the coefficient values in the 
major subniatrices, thereby offering a further reduction xn dimensionality 
by continuing to microcode or by a hardware design. If microcoding were 
to be continued, then the storage space would have to be doubled for each 
addition bit, whereas, if a hardware controller were added with a recir- 
culating control register containing the pattern sequence code, the stor- 
age array would not Increase. Such hardware control sequence code shall 
be termed program microcoding. There may be a combination of the two 
techniques that reduces the pseudoinverse dimensionality to the intuitively 
determined minimum of twice the number of sensors. 

Beginning with the 2n coefficient array storage necessary for the 
basic system, a maximal control microcoding would maintain the minimal 2n 
storage area, whereas any additional coding in the form of program 
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Values displayed are absolute. 

Signs form redundant patterns 
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pseudoinverse and can be programmed by the control cxrcuitry with a 
single bit for a sign. 


Equation (5-23) 



microcodxng would serve to swell the storage area. However, a type of 
storage for the controlled loading of the programaable filter must be 
provided and this constitutes an increased storage area even though this 
appears to be of a minimal nature due to the pattern redundances and the 
independence from the coefficient array. For the moment, further discus- 
sion of microcoding will be suspended in favor of a brief look at a stor- 
age arrangement of the coefficient values in an attempt to clarify the 
reasoning for the bias in the resultant choice of a microcoding technique. 

4. 

Recalling again the pseudoinverse M matrix and the major submatrices 
indicated in Eq. (5-23) , an apparent technique would treat the rows com- 
posing the major submatrix as words to be loaded into an intermediate 
register preparatory to programming the filter. Since there appears to 
be t^7o different words per minor submatrix, two program or address 
counters and appropriately gating logic (decoding) to the intermediate 
register would be indicated by the unit step increment change between 
word blocks in adjacent minor submatrices. Thus, a storage arrangement, 
as described, would maintain minimal 2n storage, yet ease the microcoding 
requirement by reducing the number of control sequences needed to replicate 
the pseudoinverse matrix. 

Returning to the microcoding discussion, because the storage arrangement 
has simplified the controlling requirements only the control microcoding 
will be discussed. Examining the minor submatrices in the first major 
submatrix sample, a suggested axis of rotation would be approximately 
half the rows as depicted by the double lines in Fig. 5-20. A hardware 
design might incorporate a control coimter to count to half the number of 
rows to generate an overflow which is routed to logic that converts the 
program or address counter to a down or reversed sequence counter thereby 
reversing the order of row display. The overflow pulse or charge packet 
also reroutes steering logic m the decoder to reverse the order of the 
columns thus implementing a mirror image of half of the major submatrix. 

A control counter may be designed as a ring counter or racetrack shift 
register driven by clock pulses. Such a counter is easily realized in 
CCds [11]. Having presented a technique for implementing the mirror image 
seemingly present in each major submatrix and a mode of reconstructing the 
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Figure 5-20. Pseudo Inverse Matrix Illustrating Mirror 
Images in Major Submatrices. 



rows for each sample major submatrix, the entire scheme could be depicted 
as in Fxg. 5-22. It can be seen that the last remainxng block to be de- 
scribed IS the decoder of which several methods can be utilized to the 
greatest advantage. One such method would utilize a multiplexor to 
selectively load a serial shift register in a coded sequence; another 
method would employ CCD register routing by a multiplexor in much the same 
manner as before, except that the loading would be done in a parallel fash- 
ion (Fig. 5-23) . 

A greater reduction in size might be achieved if the programmable 
filter were deleted. Instead of shifting a normalized mit charge and 
parallel loading the output with the proportional analog value (Fig. 20), 
the sensor outputs are parallel loaded into the analog storage device. The 
tapped weighting electrodes are floating-gate amplifiers connected to the 
proper sign on the differential current meter. This realization would use 
B storage devices of the length of n sensor cells, where B is defined as 
the sum of the number of distinct row values per major submatrix over the 
entire completed sample. Decoding and control circuitry would be implemented 
similar to that described for the programmable filter resulting in a greater 
savings in complexity (Fig. 5-24). 

Most attention has been directed at the circuitry for a single major 
submatrix sample. Structures which incorporate the total coefficient matrix 
will now be considered. A direct technique would attempt to process all 
the sensor output samples through a single transversal filter, reducing the 
number of filters but increasing the processing time with a small addition 
to the circuitry. Another technique has been suggested [7] which employs 
a staged approach easily seen if each major submatrix is thought of as a 
stage thereby separating the samples into a natural division. The Westing- 
house report [7] confines itself to the direct storage of each individual 
element in a coefficient array, and as previously seen, this would he an 
impractical array. Next, an example design will he given to test for an 
approximate maximum number of sensors that could be fabricated on a single 
chip containing the signal processing circuitry. 

In an attempt to generalize the design, fixed tap weight shift cells 
and normal linear shift cell shall have a constant area A. The programmable 


93 



<X) 

-p- 



I Stored Value 
7 Outputs to Decoder 
I and Steering Circuitry 


UP/DN Select Line 
Clock 


Figure 5-22 « Illustration of Counter Arrangement to 
Implement a Mirror Image Coefficient 
Matrix 





Programmable Filter 


= 2: 


Multiplexor/Storage 


Storage Array 


Enhanced 
Serial Output 


Decode and Control 
Circuitry 


Figure 5-23. Block Diagram Illustration of Microcoded Processing 
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Figure 5-24. Block Diagram Illustration of Microcoded Processing 
Array and System with Single Storage Element Detail 









tap weights shift cells, including the MOSFET structure, shall have an area 
kA, where k is some constant multiple of the area A with a value greater 
than one (k > 1) due to the added MOSFET structure. Each current sensing 
differential amplifier shall have a constant area to be designated by the 
value D, Area between shift cells in a parallel loading scheme shall be 
assumed to be negligible. 

The design examples shall be presented using total area as a criteria, 
thus asstaming that the trade-offs between circuit complexity and processing 
speed are apparent. The sensor area and input/output circuitry shall be 
Ignored because these may be chosen independent of the processing circuitry. 

Retracing the suggested designs and beginning with a parallel structure 
involving a single fixed tap weight transversal filter per row it can be 
seen that each shift cell in each filter corresponds to an element in the 
pseudoinverse column. The area of the signal processing circuitry as depicted 
by Fig. 5-15, can be expressed by 

TOTAL AREA A^^ = (rc)A + rD (5-24) 

where r and c are the number of rows and columns, respectively, of the 
pseudoinverse M matrix. The next design employed a programmable trans- 
versal filter with (rc) absolute storage values, assumed to be analog fixed 
tap weight storage elements. Assximing that the decoding and control cir- 
cuitry is negligible in comparison to the storage area (less than 10%) , the 
area (design as in Fig. 5-25) is apparently 

TOTAL AREA ^ 2(rc)A + k(cA) (5-25) 

The 1st designs involve microcoding schemes of which there were several 
variations. The first scheme illustrates the microcoding of each major 
submatrix as a mirror image (Fig, 5-26). Decoding and control circuitry 
directs the loading of the programmable filter for each major submatrix. 

TOTAL AREA A = (Bn)A + H + 4kcA (5-26) 
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represents such a scheme wxth a 4:1 enhancement where H xs defined as the 
area due to -the decoding and control circuitry, n is the ntunber of sensors, 
and B the sum of the distinct row values per major submatrix over the 
complete sample. Attemptxng to characterize the best of the microcoding 
schemes, the last design discussed (refer to Fig. 5-24) shall be represented 
by 


TOTAL ASEA A^^ = [(Bn)A + BDj + H (5-27) 

where the difference between (5-26) and (5-27) is the deletion of the 
programmable filter. This is accomplished by the utilization of the storage 
elements as the processing circuitry. 

The four design equations presented represent four different system 
implementations Tnth their consequential trade-offs. Since various con- 
siderations are encountered in each design resulting in different variables, 
a cross comparison would be difficult to make. However, a tabulation of the 
design equations is presented in Table 1 as well as an estimated area for 
a given number of sensors capable of being processed in a 4:1 resolution 
enhancement. The area estimations are based on intuitive trends evidenced 
in examples of each design conducted with a small number of sensors and on 
an assumed constant cell area. It can be seen from Table 1 that the micro- 
coding scheme demands the least area for the number of sensors. 

If larger arrays are to be processed than can be easily accommodated 
in a single device, then another type of staging would be required. Attempt- 
ing to realize the "smart sensor" concept, the combination of the microcoding 
technique with the staging approach depicted in Fig, 5-27 could make such a 
realization possible. The additional circuitry to process the overlapping 
external device sensors is easily justified by the trade-offs in information 
processing capability versus practical implementation size. Such a combina- 
tion of techniques should remain useful regardless of future technology 
density improvements. 

Error Considerations 

Having seen several practical implementations of the resolution en- 
hancement scheme in designs realized by charge-coupled devices, attention 
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is noTr focused upon the system's liabilxties, particularly in the form of 
error. Revxewing the previous dxscussion on the error inherent in McRee’s 
technxque, the error shall begxn with the error dxsplayed in the system- 
operating algorithm. 

From (5-9) , it can be seen that any inxtial error would propagate and 
accumulate through to the completed sample. The boundary condxtions xmposed 
by McRee to facilitate matrxx inversion was removed by Henderson in his 
algorithm utilizxng a least squares estimatxon technique to generate a 
pseudoinverse which contaxns the system design values. Another xmportant 
limxtation removed by Henderson [3] was the accountability for a noxsy 
input whxch although assumed to approach zero for the realizatxon has 
practxcal signxficance. However, xf noise is present in the input, it 
could be characterized xn the system input and will not be consxdered here. 
Henderson has experimentally determxned the error for his algorxthm to 
range from maximum of exght percent to a neglxgible min imum . However, the 
system on which the measurements were made xncorporated analog-to-digital 
converters to quantize the sensor outputs before being fed xnto the proc- 
essing system. Henderson has noted that by increasing the number of 
quantization levels the accuracy xmproved signxfxcantly. The implxcatxon 
of this apparent trend is that quantizatxon error is a sxgnificant con- 
tributing factor to the overall system inaccuracy. If the estimatxon error 
xs consxdered to be xnsxgnxficant, then quantizatxon error xs the only 
remaxnxng error source in the system algorithm. As mentxoned before, if 
CCDs are utilxz,ed as a transversal filter, then analog sensor output is 
processed thereby reducing system error to the practical limitations of 
the device. Elimination of the conversion units (a/D) has two definite 
benefxts, namely, to reduce system complexxty and to signxfxcantly improve 
system accuracy. Wxth thxs xn mxnd, promxnent CCD limitations and error 
sources should be considered, and, since discussions of error in CCDs have 
been published (17, 18, 19, and 20), only a brief survey will be included 
here. 

Commencing with the most signxf leant source of error, transfer xnef- 
fxciency s, it can be seen that, as a packet of charge xs transferred 
through htmdreds of cells, that some charge carriers would remain trapped 

blank not mMm 
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in substrate imperfections or be lost to recombination centers. As can 
be expected, the transfer inefficiency e would be a function of clock 
frequency and also of device geometry; therefore, each device designed 
would have a different value s with current s approaching 10“^. Another 
source of error would be that generated by nonlinear it ies in the charge 
injection and charge sensing, with the last prominent error source to be 
that of thermal generation. Injection and sensing nonlinearities can be 
treated in various ways as a trade-off between complexity and size of the 
circuitry versus nonlinearity. Thermal generation is also termed dark 
current and is most prominent when low clock rates, high temperatures, or 
numerous shift cells are encountered. All the prominent error sources 
can be compensated for by several design techniques with the accompanying 
trade-offs. There are other error sources beyond the device level, namely 
those encountered in device fabrication, such as photolithography resolution, 
all of which should be reduced \dlth evolving technology. 

A more definite characterization of error should be undertaken after 
an experimental device has been fabricated. 

Conclusions and Recommendations 

Several designs are presented using CCD technology to implement an 
unique enhanced resolution algorithm. System errors are expected to be 
largely dependent upon device error inherent in the fabrication except 
for sensor and algorithm error. It has been shoxm theoretically that 
CCD technology can produce a significant reduction in processing circuitry 
complexity (e.g. elimination of A/D conversion). A further savings could 
be realized by use of microcoding techniques as shoxra by Table 1. The last 
design presented CA^^^) indicates the best circuit economy even with conser- 
vative assumptions. Processing speed is a design trade-off between resolu- 
tion enhancement, target or sensor maximum velocity, and circuitry complexity. 
However, processing time is expected to be in the tens of microseconds for 
a single sample conversion. Experimental design and fabrication of devices 
should be undertaken to help define the best nximber of sensors for a building 
block or staged approach. Complex trade-offs between number of leads, re- 
dundant circuitry percentage, and other practical considerations make 
experimental work desirable. 
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TABLE 1 


Designai-ed 

Design 

Design Equations 

Estimated Parameters 
for Five Sensors 
(square units) 

Estimated Parameters 
for Three Sensors 
(square units) 

^ti 

(rc)A + rD 

900A + 36D = 1044 

144A + 16D = 208 


2(rc)A + (kc)A 

1800A + 25kA = 1850 

288A + 91cA - 306 

^t3 

(rc)A + D 

900A + D =904 

144A + D =148 


(Bn)A + BD + H 

70A + 14D + K = 378 

24A + 6D + H = 96 


Assumptions : 1) 

CJ 

II 

> 

2) 

H - (Bn)A + ED 

3) 

k = 2 

4) 

A = 1 


Table 1. Compilation of Design Equations and Estimated Results for Three and 
Eive Sensors with Assumptions of Support Circuitry Size. 











The sensor array xs assumed to sense and retain a sample of a displayed 
energy density field (e. g. light). By varying a threshold such that the 
value above the threshold is processed, a vertical binary mosaic represent- 
ing the energy density can be formed. Preliminary studies showed that in 
this manner gray-level information could be retained, but to an unknown 
degree of resolution. A vertical binary mosaic using a previous example is 
displayed in Fig. 5-28. However, it can be seen that a large degree of error 
is involved which is suspected to be due primarily to quantization. 

With the ability to preserve gray-level information and the inherent 
angle information due to the sampling scheme and motion, another form of 
information may be available. By properly correlating the information 
input into the system, a pseudo-dimensionality may be achievable giving 
rise to "depth" infortiation. Such speculations are reserved though future 
work. 

In either case, it can be seen that the processing of analog informa- 
tion is highly advantageous, particularly from the low error viewpoint. 
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Figure 5-27. Illustration of Staged Staggered Sensor Column Array to Show 
Formation of Larger Arrays. 
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Abstract 

The application of CCD (Charge-Coupled Device) processors to pattern rec- 
ognition and control systems is suggested in this paper- Specific implementa- 
tions Tor control systems are presented for linear time invariant, linear time 
varying, and nonlinear systems. Example structures are also presented for pat- 
tern recognition applications. Estimates of processing times are provided 
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Introduccion 


Rect^ntly, a new form of charge transfer device called the CCD (Charge- 
Coupled Dev3ce) has been developed which processes discrete analog samples of 
data. The inherent manufacturing simplicity, small size, and low power allow 
the use of many stages in a single unit. This, coiipled with their fast speed, 
makes practical the processing of extremely larger quantities of data in appli- 
cations such as image processing. Although devices are not yet commercially 
available, families are expected to emerge soon, thus, it is timely to consider 
their application in engineering systems 

The initial discovery of CCD's was announced by Bell Telephone 
l.ahor.sl on es on May 9, 1970 [1] Since then, commercial devices for optical 
signal iJctection, such as discrete element sensors [2], ond for digital memo- 
ries (3], have boon marketed. 

A charge-coupled device is a semiconductor component [4] capable of stor- 
ing quanlities of charge (called packets) and transferring Lliem between adja- 
cent cells. This, in effect, becomes a discrete analog delay line or shift 
register The addition of nondestructive readout taps, along with tap weights 
and on-chip summing, provides an analog processing capability of umoue 
potential . 

Another charge transfer device, which was the forerunner to CCD's, is the 
Piukot Brigade Device (BBD) [5]. The BED consists of a series of field effect 
transistors in which charge packets are transferred from one cell to another. 
Tnc bed’s ire of less value for sophisticated signal processing and, thus, will 
not be considered in the sequel 

For a detail c^d explanation of CCD characteristics and operation the 
render is referred to [6] Essentially, they are composed of a substrate of 


119 



semicondvic. Lor material on which a thin insulation layer is placed. The cells 
in which the charge packets are stored are formed by depositing metal elec~ 
l nodes onto the insulator. This forms capacitors which operate analogously to 
the (.apaciLors and switches shown in Figure 1. When any switch closes, charge 
IS transferred to an adjacent capacitor, and then the switch is opened. With 
proper synchronization between switches, charge transfer can be controlled. Of 
course, this representation is crude; but it is a convenient approximation of 
opera L i on 

Tn physical electronic terms, when the electrodes are properly biased 
With respect to the substrate, potential wells are formed in wnicb the charge 
IS stored The charge transfer action is produced by charging the potentials 
associated with adjacent cells Chax'ge is injected into the input cell through 
a rnnvcnLional junction diode Operation of this diode in conjunction wiLh the 
first cell allows sampling of the analog input signal, yielding a discrete 
analog representation. 

As might be expected, charge cannot be iransfeired between ^cells without 
loss, since a fraction of the charge will be left behind, thus causing a reduc- 
tion in l !ic SI gnal-to-noise ratio. Typically, the transfer inefficiency e is 

-h -5 

on the order of 10 to 10 at 1 MHz and increases rather rapidly at higher 
f reciuenc i <‘s . 

i’lic unique discrete analog signal processing capaoiliLies of CCDs may 
make praclical pattern recognition systems which, until now, have been only 
tlicoi c"t ii. t 1 possibilities because of the extremely large quantities of data to 
be j)i iK ( "-sed Also, control system applications appear to be practical in 
systems of high order, such as where Kalman filtering is required. 
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Basic Structures 


riic-* most basic structure is a serial in/serial out (S3 /SO) discrete 
an.ilog c3e]ay line, which is shown in Figure 2a. The data are transferred fiom 
left to right at a rate determined by the clock, with a period T. The maximum 
time for a sample to be in residence (delayed) is NT. Delays of 0.1 to 3 
seconds at room temperature are reported in the literature and calculations 
indicate that much longer times are possible if the tempeiature is reduced. 
Tliere are many applications for this basic structure. It is also used as a 
building block in more complex components 

A serial in/parallel out (SI/PO) structure is shown in Figure 2b, The 
input signal is sampled and shifted as in a SI/SO device, but the content of 
each cell is available on a nondestructive readout basis. If tap weights are 
used in conjunction with current summing, it is possible to implement a number 
of useful opc^rations which are considered later, 

riic parallel in/serial out (PI/SO) conf iguratioa is shown in Figure 2c. 

It can be u.sed for applications such as multiplexing. 

rhc slructure of Figure 3 is likely to become a basic building block in 
tile lu ’it future. shown, it consists of a Sl/SO structure with tap weights 

and T sunimer as mentioned above, all on a single chip Tiie current summing is 
tasl^^ )nd accuritely accomplished because of the basic way the structure oper- 
it(S. It operates as a nonrecursive or transversal filter, whose output at the 
n^^’ clotk period is the weighted sum of the previous k samples, 

k 

g(n) = I h X n > k (1) 

1 n — 1 - 

1 = 0 

Foi c' iinple, if the tap weights h^ in (1) define samples of the time inverse of 
a desired signal, then the CCD processor of Figure 3 is a matched filter. Or, 
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if Che tap weights are set equal to samples of a given weighting function, then 
the structure can perform a discrete convolution. Finally, vector inner pro- 
ducts can be computed by reading the output only at n = k, so that 

k 

g(k) = S h^ (2) 

1 = 0 

By cliuiging the order of the weights in Figure 3, the more familiar form 

k 

g = I h X (4) 

j=0 

IS obtained, which can be recognized as a linear discriminant function [7], a 
very useful tool in pattern recognition. 

Fixed tap weight transversal filters are also being used to implement 
frequency domain processing. For example, by employing the Chirp z transform 
18], (.CDs can be used to implement discrete Fourier transforms which compute a 
500-point power density spectrum [9]. By using CCD Fourier transfoim proces- 
sors, convolution of two arbitrary waveforms can be accomplished through fre- 
quency clomdin multiplication. 

Another form of CCD processor is under investigation which could prove to 
be veo valuable to the structure of future instrumentation systems. As shown 
in Figure 4, this filler allows the tap weights to be programmed electrically, 
thus making it applicable to adaptive control, trainable pattern classifiers, 
etc. riu variable tap weight transversal filter is difficult to implement and 
will piob.'bly not be available for some time. Nevertheless, it could be very 
import int to the development of new system architectures and so deserves atten- 
tion ( ven now. 
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Control System App3 icabions 


Control systems can use CCD devices to make modeling calculations or for 
:oal 3 /cing required transfer functions [10]. State space models are an example 
where matrix-type calculations are required. Suppose, for example, a linear 
time invariant system containing a Kalman filter is to be implemented Then, 
(a]rulation of states at time k + 1 based on states at time k must be made 
using the state tiansition matrix. The unforced system differential equation 

IS 

X = Ax (5) 

for which the solution is 


, , A(t-to) s 
x(t) = e xUo) 

For the discrete case of interest here, (6) becomes 

x[(k + 1)T] = 0(T)x[kT] 


( 6 ) 


(7) 


wliei e X IS an M-dimensional vector, A is an nxn matrix, T is tlie sampling 
period, IS the nxn state transition matrix, and k is a positive integer. 
As is well known, (7) takes the general form 


x[(k + 1)T] 


‘^11 *^12 
'^21 *^22 




x[kT] 




nn 


(8) 


Uiere rlie elements of 4' <ire constants for the time invariant cases being 
i oils] dc red. 
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Imp]einentation of equation (8) is shown in Figure 5, Each transversal 

falter is equivalent to the basic structure of Figure 3, where the values of 

the tap weights are equal to the row components of \the 0 matrix, i.e., for the 

top filler, h, equals , h, , equals etc. This assures that the first 

k 11 k-1 12 

component of the state, Xj^(kT), is fed in first, x^CkT) is fed in next, etc. A 
typical cycle time might be ti + 2 clock periods to obtain the state x(k + 1)T 
in storage in the Pl/SO register from the initial value x(kT). 

The reader will recognize that this implementation can be used to simu- 
late Jlie homogeneous solution for any dynamic system. A complete simulation 
would include another matrix driven by the input plus appropriate summation, 
etc 

An alternate approach using transversal filters for control and simula- 
tion makes use of difference equation formulations. The general n-th order 
linear time-invariant difference equation is given by^ 

y (k) = b u(k) + b,u(k - 1) + • * ■ + b u(k - p) - a y (k - 1) 
of pi 

(9) 

-a y(k - 2) - • • • - a y(k - n) , k = 1, 2, • • * 

i. n 

where y{k) is the output at time t = kT and y (k - 1) , y (k - 2) . . are the 
output values at (k - 1)T, (k - 2)T, etc. Similarly, u(k) is the input at 
t = kT and u(k - 1), u(k - 2) . . are the input values at Lines (k - 1)T, 

(k - 2)T, , and the and are the system coefficients. This is a 

recuisive equation involving both the input and past values of the output. 

Ih cause of the need for additional on-chip amplifiers, CCD recursive filters 
tn.iy not be ns cost effective as transversal filters Thus, the suggested 
impl c nienta Lion of (9) makes use of external feedback paths aiound nonrecursive 
filters, 3s shown in Figure 6. 
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CCDs appear to offer much promise for real time simulation of large 
linear time invariant systems, because tliey tan process l<irgt> quantities of 
discrete analog data very rapidly. For time varying and nonlinear cases varia- 
ble lap weight devices would seem to offer promise Structuresrequired for 
their implementation would be similar to those already presented, except that 
the filter in Figure 4 would be used. 

P attern Recogniti o n Appl i_< a_^i ons 

Jlie pattern lecogniLion system to be considered is shown in Figure 7. 
riie input waveform, s(t), to be identified as belonging to one of C different 
daises, k = 1, 2, . , C, is sampled to form a vector z as 

z = (z , z , • • • , z )*■ (10) 

12 n 


and ftd to the Feature 1> tractor. 

Die purpose of the Feature Extractor is to measure significant piopirlies 
of me vector z so that the Classifier can make accurate decisions This 
ji'ipliis a d.it a rodiuLion transformation which removes much of the redundant 
1 nrorm it 1 on wliile compressing relevant detail into as few feature measurements 
IS possible. Only linear transfonnations are being considered, so the Feature 
I'xtrartor c<m be modeled by 


X = Az (11) 

wbc 1 o 7 1 '^ n > 1, the trans formati on matrix A is d x n and the feature vector x 
IS d ' 1, with d < n. For eximple, among the t rans foi mat i ons that could be 
usc'^ci lie the discrete Foui ler and Hadamard ir asfoiins. The feature vector 
servos as nqiut to the Classifier 

bnder the assumption that similariti among different feature vc-*ctors can 


ORIGINAi; PAGE IS 
OP POOR QUALITYi 


125 



bo represented by some distance criterion, the function of the Classifier is to 
partition tlie feature space such that feature vectors from the different pat- 
tern classes lie in distinct hypervolumes and can thus be discriminated 
!lic extent to which this may be done determines the error rate for the system 
Once again, the discriminant functions which make up the Classifier are 
constrained to be linear, such as 

g^(x) = w^x + 1 == 1, 2, - • • , C (12) 

wh(>ie w = (w , w , • • • , w ) is the weight vector and w is the con- 
1 liiji Id xo 

slant a-,sociated with the i*"^ class. This still leaves a large variety of pro- 
bit ms for which CCD pioccssors will be applicable. 

To illustrate CCD processor organiaations which may be applicable in pat- 
tern itcogmlion systems, consider the following hyperthetical problem [11]* 

1 Number of sample points, n = 200 

2. Number features, d = 20 

3, Number classes, C - 60 

Ihe feature extraction tnnsfoimation to be used was given by (11). For 
I onvem c>nt f , let w^^ = 0 so that the Classifier consists of the discriminant 
f mu t I ons 

g^(x) = w^y 1 = 1, 2, - • • , 60 (13) 

F 'ttiiU'ion to the case w t 0 con be done easily 

1 0 

One way xn which to implciiient (11) and (13) inv'olves a structure xn which 
all disci inunnnt functions are evaluated s imul taneousl y , without the need for 
siparaip feature, extractic^n Such a reali.iation is suggested bv a 
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I wy-di men'? lon.il CCD mnLr^x array [12]. First, substitute (11) into (13) to 


obtm n 


g = WAz = Qz (14) 

wheie g = (gj^(x), ' ' ’ > W = (w^, • • • , and Q is a 

60 X 200 matrix 

Fit'uie 8 shows the use of a bl/PO CCD siiift register to obtain samples of 
the waveform to be classified. The Q matrix coriesponds to the analog weight- 
ing conductance array, represented by dots in the figure. The PI/SO CCD shift 
rc--gister is used to shift out the discr msinant values to a serial comparator, 
which cletcniunes the maximum discriminant function and thus makes the classifi- 
(at ion. If It is assumed that the CCD registers are operated at 1 MHz, then it 
IS fstjmitc'd that a decision can be made in about 600 ps [ll]. 

AnoLlier possible system structure is sliown in Figure 9 which utilizes a 
ser 1 cs-jiaral lei organization involving vaiiable tap weiglit CCD lorielators, 

I e id only memories (ROMs) and digital-to-nnalog converters (DACs) Again, the 
0 in.it rix of (14) is used but now is stored in the ten Q matrix ROMs. Conceptu- 
.illy, <a<h ROM tont.iins six lows n.sving 200 data each. First, the r.ninple func- 
tion to be classified is acquired and then one row at a time of the ten ROMs is 
loidcci into the correlators. The operation performed is an inner nioduct of a 
given I ow of the t) matrix with the z vector obtained f 1 om s impl mg the input, 
.since no l.igs are involved. Ten discriminant values «re ohtainc-d in parallel, 
remiiiim; a total of six cycles of operation for the arbitrary number of DACs 
shown in the figure Again, using a freouency of 1 MHz for the CCDs and me*!- 
]i«'nsive (slow) DACs, a decision will reauire about 13 tns [11] 

Jneie Ml, of course, many variations of Figure 9, for example, in which 
a different nuir.ber of i3ACs are used In 'doition, it may pi eve advantagnus in 
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‘.onif c ises to provide separate structures for implementing the Feature 
! XI rat tor and Classifier, rather than combining them as in (14). The straight- 
forward use of a number of fixed tap weight transversal filters, such as in 
Figure 5, is also likely to be of value in some rases. 
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THE APPLICATION 


OF CHARGE-COUPLED DEVICE PROCESSORS 
IN AUTOMATIC CONTROL SYSTEMS 

E. S. McVey, Sr. Member, IEEE E. A Parrish, Jr., Member, 3 FEE 


Abst ract 

The ajipli ca tioii of CCD (charge— coup] ed device) processors Lo automatic 
control systems is suggested. CCD processors are a new Conti of sem i conthu Lor 
component with the unique ability Lo process sampled signals in an tntlog 
basis Specific implementations of controllers are suggested for linear Lime 
invaiiant, time varying and nonlinear systems. Typical processing Lime should 
be only a few microseconds. This form of technology may become compelilive 
with itn cropiocGSso) s and minicomputers in addition to suppltniu nl i ng llicin 
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THE AITLICATION 


OF CHARGE-COUPLED DEVICE PROCESSORS 
IN AUTOMATIC CONTROL SYSTEMS 

E. S. McVey, Sr Member, IEEE E A. Parrish, Jr., EJember, UE.E * 


Introduction 


Tlie devel oj)i.ienL of CCD (charge-coupled device) processois offers rontiol 
eugineeis new possibilities for implementing controllers. Charge-coupled 
devices are unifiue :n that they make possible processing of discrcHe .inalog 
data Their discovery was announced in Hay of 1970 by the Bell Telephone 
Laborntories [1> 2, 3], and a few commercial devices are available. [^ ] 

Althougli It IS not yet practic.il to use CCDs in production equipment, ert ept 
for appli cat ions involving memory and discrete image sensors, it is time to 
establish a foundation for their use in contiol systems. It is ii.iporl.int that 
dovjie researcliers and manufacturers be made aware of the needs of control 
technology. It is the puipose here to suggest possible uses of CCD piiucsors, 
delineate some of the expected problems and suggest possible .aieas of 
e>pl oi ntion. 

A CCD IS a seiin cunduc tor romponentl^] capable of storing (juniltiies of 
charge called packets ind moving them practically intact from one storage loc.i- 
Lion to inollier The packets represent the magnitude of sign.il sniples If 
the movement takes place at regular tune intervals, a discrete dcl.iy lane 
1 C L ton IS aclneved 

The CCD js a im.inber of the class called charge transfer devices of which 
the "bucket brig.ade device" (BBD)[6] is also a member The BBD is oldei but of 
1 ( ss polenti il value for control system signal processing than the CCD.t^] A 
auunber of papers are .ivailable which describe in detail tlie eba nctci ist i < s of 
CCDs, for ex.imple, 1.^1 They consist essentially of a substrate of scmic omluc- 
tor ii.alerial, sucli ns n-type on which is placed a thin layer of insulation 
Cells ire then foLiiied by depositing metal electiodes on the insulating mate- 
rial. Biasing the .^Icctiodes with respect to the substrate creates a potintinl 
well in Llie semi n>n-hn t or material. Cliarge injection is nccoinplisbed wi Lli a 
conventional PN junction Charge may be transfeired between ndjaient cells by 
appiopri ately jiulsing the associ.it cd electrodes. Although the tr.insfer bt'Lwecn 
cells is not perfe. t, it is idequate for many appl i c.n tions . The tr insfer Jnef- 
ficieiiey c is on the order of 10 ^ to 10 **.[9] The r. depends on clock f i o- 
cjuency, as one wovil d c pect Present opcr.ation of these devices is usually 
restricted to less than 10 MHz. Tr insfer efficiency is important to .in ilog 
d It .1 processing, bee luse sjgn.il It'vels cannot be reestablished as in digit.il 
processing. 'Iho th< rmal gener.itioii of electron-hole pans is a serious souice 
of distortion, exj-eciilly at elevated temperatures. This restricts stoi'igc' in 

Both with tile School of Engineering and Applied Science, binvc>rstty of 
Viiginia, Chariot Le >vi 1 1 e , Virginia The research on which this pajicr is li.i ed 
CIS su'iporied by tm N \SA I inglcy Rese.irch Center on Gr.ant No ISG-lOb? ind 
Gi irit No. 1151. 
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a cell at room temperature to times less than one second. Storage time c<jn be 
increased approximately by a factor of two for every 10*C decrease in 
temperature. 

This technology has led to the development of discrete element sensors 
and TV cameras that are now available commercially . t 10] It is anticipated that 
memories of very high densities will be developed. SmaJl inomories are now 
available commercially. Discrete analog CCD processors are not yet widely 
available, but families are expected to emerge soon. 

Digital computers, especially small ones, are becoming widely used for 
control purposes. This seems to be a logical application also for CCD hard- 
vtfare Different imp] oinentation possibilities are suggested in the following. 

It IS emph.isiaed that these ideas have not been proven cNperimc'nt illv , h< i luse 
It IS not yet possible to obtain the devices However, lliey sie logii.il c>\li3— 
polations based on successful experiments and on anticipated device 
developments 


Fund amen t al Con fig u r ati on s 

Several CCD sum Lures are under development which are considercnl b isic 
building blocks. By using these devices with some additional ciicmiiy, it is 
possible to impleraeiir linear L ransformati ons such as convolution, coiicl.uion 
and Fourier trnnsfoiras. 

fhe three basic building blocks are Serial Tn/'^eiiil Out (M/'iO) 
Registers, Serial In/Pirillel Out (Sl/PO) Regisleis md Pirallcl In/'.iiii! Out 
(Pl/bO) Regi sters. f i 1 ] Of these, the SI/SO is the simplest, com- i si i ii); of n 
linear .irray of CCD lells Depending upon the number of cells uid the <lmk 
) ite, v.uious delays may be achieved This structure is fiiiidumiit il 1 1 ' UD 
iitc mo lies. 

i’he bl/PO block has nomlesl i uctive taps along the delay line. By i>iovuU 
mg weights on these Laps and using curient summing, a device such .u> Oiown m 
Fig 1 IS obtained. If the tap weights aie made equal to the corapoiunts of a 
vector h = (hj , hp , . , h^j)t and a vector u - (uj , up , . . , 'hj)^ is 

louled into the sliift register, then the ciicuit will coimniie the. mm r piodiut 

y = h * u (1) 

Mitthed filteiing ran also be sc c ompl i slied by setting the Lap wcMglHs 
equal to samples of (he time inverse of Llie signal to be detccti'd At the 
clock time the output is then the weigliLed sum of the pievious nmiles 

P 

y(n) = i: h u , n > p (2) 

1=0 

The fixed lap weight structuie in Fig. 1 is only a little niuie cuinpli- 
tated than the basic bl/PO building block. Bet.mse of tins and the fart thit 
It nnplei.Knts a transversal or nonrceuisivo filler, it Loo is likely to befoine 
a b.isic building block in the neni future. 1^2] 

Some woik has been done on strut lures siinilar to that in Fig 1 in which 
the tap wc’ights are variable Ibis would .illow convolut ion or cntic’l i- 

1 1 on of two irbili ny I'lvefoins in addition to finding applit itioiis in i inu* 

V iryiiig .ind adiptivo control ‘-vstems fo dite, successful fabi nation of 
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variable tap weight transversal filters has been somewhat limited compared to 
those in which the tap weights are fixed and represent a considerable chal- 
lenge 115] Because of their signifjcant potential in control systems, struc- 
tures will be suggested to demonstrate the need for continuing the tievolopmont 
of these devices. 

The PI/SO block is, of course, similar to the Sl/PO block, except for 
1 n te r— chs ng ing the input and output Tlie register is loaded by synchronously 
sampling all of its inputs, ihis configiiration is shown in later figures. 

Ac this point It may be well to note the conditions under which CCD pro- 
cessors might become practical for use in control systems. Classrc.nl ly , the 
control engineer would implement a transfer function with passive components 
and nn amplifier. One condition Chat would make the CCD system piefeiable to 
the classical approach for a linear time invariant system will exist when the 
fixed tap weight transversal filter of Tig. 1 costs less than the controllers 
presently being used Otlier conditions include the potential for lecpiiring 
less space, power, volume and increased reliability. Timing functions not 
shown in the diagi..ra increase the costs of the CCD controller. A bigger 
piyoff IS avail iblc in more complicated systems winch require parnmelei tlianges 
not cMsily implemented With classical methods. In this case, the (CD control- 
ler competes with a foim of computer. 


_^n^ol S^st_enis P aiiiples 
Li_ncar_ ^fime Invariant Sjy’steins 

Ihere aio several anproaches for modeling automatic contiol sywioms which 
are suitiible for (.( D !mplein[>nt.iti on. In the following, discrete e onvt>ln( ion 
md dificience c'cpi.ition models will be lonsidercd, although si lie sp n i> iiuulels 
tan be used, also 

As a piactjcal exiinple, the following weighting function (unnns in 1 1 r- 
framc component) for a commauiil aircraft will be consideied Tor the pilch 
rate inpiii , it is given by 

h(t) = - 5 35e”^‘®^ 

-u .r 

+ e •^'-(4 8cos4.3t + 6 9sin4.3t) 

Ihe output y(t) may be expressed as 

y(t) == h(r)u(t - r)dT ' (4) 

which has the discrete form given m (2) For this case, the values of h (o 
use in Fig 1 for d = 200 can be obtained from (3) over a 10-second lime intt*r- 
val. It would be practical lo use oven more v. lines of h. For eiamjile, a bOO- 

point Ct D Fouiiei li aisform proc<*',sor is being developed t , and a bOO-pnint 

CCD power density spectrum analjxer In', been tested. 

A typical operation cycle oi tlie CCD processor might bo 6 mic to ,ei mids 
> a now valuG cif y (n) is aviil ible every 1 = 6 inic rciseconds) wlu>re I 
ml ( 1 o‘,r cond is rc'quiieci to shift .mother s 'nple of u(t) into the shift i t'g i '.i er 
.ind j nil I 1 o.,i t O'liis is i (.-qin i ed lor the siii>'inii mn .md oper.Uion of the 'cio 

Older hold devu c' lliis is ninth f isls r tli la could bo obtained with ,m oidiniiy 

digit.il computer. 

An .slturnative v ly to rc.ilize this controller would be to disi i < i i /e i he 
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transfer function using Tustin's approximation, for example, and then convert 
to a difference equation formulation. In general, the result would he a recur- 
sive difference equation involving both the input and p.ist values of the 
output. A recursive impleinentation is proposed in Fig. 2. It has been sug- 
gested [12] that CCD recursive filters may not be as cost effective as transver- 
sal filters because of the need for additional .implifieib on the chips Conse- 
qucntly, the following example makes use of external feedback paths around mm- 
recursive filters. 

The general order linear time-invariant difference equation is givc^n 
by 


y (k) 


bQu(k) + bju(k - 1) + . . . + bpu(k - p) - ajy(k - 1) - a 2 (k 

~ - - ■'^ny(k - n), k = 1, 2, . . 




( 6 ) 


where y(k) is the output at time t = kT and y(k - 1), y(k - 2) no the 

output values pieviously stored in the top shift register of Fig. 2. Sniii- 
larly, u(k) is the input at t = kT and u(k - 1), n(k - 2) , . <iie llio input 

values picvLously stored in the bottom shift legister, and the toeffn'ionis 
and bx are the tap weights shown. Neglecting initial conditions, oper it mn is 
begun by clearing both registers. 'lliercaCter , values of y(kf) are gem i ii cd it 
each clock time 

\n anticipated s impling interval for a system like tins might be on I lie 
order of 10 microseconds wheie 6 microseconds are requ n ed for one c>c le of 
opiration of the two ti iiisversal filters and 4 niicrusetonds aie i equ i i cd for 
the final summation tliat provides y(kX) Such a small inLeival in most syst.ms 
would make it difficult to detect any difference from a conLiiuious c i e Tt is 
iiiiporLant to note that the iiiinimum sampling rate is indcpenck'iit of the oidcr of 
Llie equation, which is opposite the case where digital eumputc-is are u:.ed 

It seems ic isonable to assume that this CCD implcmc-ntaLioii will he cost 
effective in the near futuie only for controllers of high older. In i i s 
where filtering (e.g , Kalman filtering) is p irt of the controller, the older 
of the system will dcteinnne the equation order to be impleiiu nl c'd With (CD 
implementation it would appear that there is no need to obtain low oidcr 
representations. 


S ys t eins i n wh i ch_ Par a meters Va ry 

In nonlinear systems or time varying systems, it is necessary to vary the 
tap weights as indicated by Fig. 3 Since the CCD control loi cycle l i me is so 
much smaller than the lesponse time of typic.il control systems, it ->1 c-ms rea- 
sonable to treat paiameter varying problems as slowly vaiying linear systems. 
Thus, a microcomputer or other computer would calculate the tap weights at a 
relatively slow rate for the CCD controller. 


Cone 1 us i ons 

ihe possibility of applying CCD tcicbnology to system coiiH oilers nid 
tillers has been suggested. This technology would seem to be most ipplii ililc' 
to large systems or to appli c ations requi i ing faster openlion thin d i >' 1 1 a 1 
coi.ipui cr s can provide Although Ihe components lequirc'd to jmnli."ic'ni the' 
structure's shown are not readily available at this tmto, it is ant i c i n 1 1 1 d thit 
they soon wi 1 1 be. 
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Li<9l: of Captions 


Fig. 1 
Fig 2 
Fig 3 


L] near Fixed Parameter System Using Discrete Convolution 
Implementation of a Recursive Difference Equation 
Linear Time Variable Parameter System Using Discrete Convolution 
with Variable Tap Weights 
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